Create a resource instance

Right-click Business Flow under Data Development, and select Create Business Flow.

Right-click Resource, and select Create Resource > JAR.

Enter the resource name in Create Resource according to the naming convention, and set the resource type to JAR, and then select a local JAR package.

Click Submit to submit the resource to the development scheduling server.

Publish a node task.

Create an ODPS MR node

Right-click the Business Flow under Data Development, and select Create Business Flow.

Right-click Data Development, and select Create Data Development Node > ODPS MR.

Edit the node code. Double click the new ODPS MR node and enter the following interface:

jar -resources base_test.jar -classpath ./base_test.jar com.taobao.edp.odps.brandnormalize.Word.NormalizeWordAll

When one MR calls multiple JAR resources, the classpath must be written as follows: -classpath ./xxxx1.jar,./xxxx2.jar, that is, two paths must be separated by a comma (,).

• Node scheduling configuration.
• Click the Schedule on the right of the node task editing area to go to the Node Scheduling  Configuration page. For more information about node scheduling configuration, see Scheduling configuration.
• Submit the node.
• After completing the configuration, click Save in the upper-left corner of the page or press Ctrl+S to submit (and unlock) the node in the development environment.
• Publish a node task.
• For more information about the operation, see Release management.
• Test in the production environment.
• For more information about the operation, see Cyclic task.

PyODPS node

The PyODPS node type in DataWorks can be integrated with the Python SDK of MaxCompute. You can edit the Python code to operate MaxCompute on a PyODPS node of DataWorks.

The Python SDK provided in MaxCompute can be used to operate MaxCompute.

Create a PyODPS node

Right-click the Business Flow under Data Development, and select Create Business Flow.

Right-click Data Development, and select Create Data Development Node > PyODPS.

Edit the PyODPS node.

1. MaxCompute portal

   On DataWorks, the PyODPS node contains a global variable odps or o, which is the MaxCompute entry. You do not need to manually define a MaxCompute entry.

   print(odps.exist_table('PyODPS_iris'))

2. Run the SQL statements
   PyODPS supports MaxCompute SQL query and can read the execution result. The return value of the execute_sql or run_sql method is the running instance.
3. Configure the runtime parameters

   The runtime parameters must be set sometimes. You can set the hints parameter with the dict parameter type.

   o.execute_sql('select * from PyODPS_iris', hints={'odps.sql.mapper.split.size': 16})

   After you add sql.settings to the global configuration, the related runtime parameters are added upon each running.python.

   from odps import options
   options.sql.settings = {'odps.sql.mapper.split.size': 16}
   o.execute_sql('select * from PyODPS_iris')  # "hints" is added based on the global configuration. 

4. Read the SQL statement execution results

   The instance that runs the SQL statement can perform the open_reader operation. In this case, the structured data is returned as the SQL statement execution result.

   with o.execute_sql('select * from dual').open_reader() as reader:
   for record in reader:  # Process each record.

   In another case, desc may be executed in an SQL statement. In this case, the original SQL statement execution result is obtained through the reader.raw attribute.

   with o.execute_sql('desc dual').open_reader() as reader:
   print(reader.raw)

You can configure system parameters as following:

You can configure user-defined parameters as following.

1. Node scheduling configuration.

   Click the Schedule on the right of the node task editing area to go to the Node Scheduling Configuration page. For more information, see Scheduling configuration.

2. Submit the node.

   After completing the configuration, click Save in the upper-left corner of the page or press Ctrl+S to submit (and unlock) the node in the development environment.

3. Publish a node task.

   For more information about the operation, see Release management.

4. Test in the production environment.

   For more information about the operation, see Cyclic task.