Resources

Resources commonly apply to UDF and MapReduce on MaxCompute.

You can use list_resources to list all resources and use exist_resource to check whether a resource exists. You can call delete_resource to delete resources or directly call the drop method for a resource object.

PyODPS mainly supports two resource types, namely, file resources and table resources. The following describes the two resource types.

File resources

File resources include the basic file type, and py, jar, and archive.

Create a file resource

You can create a file resource by specifying the resource name, file type, and a file-like object (or a string object), as shown in the following example:

resource = o.create_resource('test_file_resource', 'file', file_obj=open('/to/path/file'))  # use file-like objects as file content
resource = o.create_resource('test_py_resource', 'py', file_obj='import this')  # use strings as file content

可以通过 temp=True 创建一个临时资源。

resource = o.create_resource('test_file_resource', 'file', file_obj=open('/to/path/file'), temp=True)

Read and modify a file resource

You can call the open method for a file resource or call open_resource at the MaxCompute entry to open a file resource. The opened object is a file-like object. Similar to the open method built in Python, file resources also support the open mode. For example:

>>> with resource.open('r') as fp:  # open a resource in read mode
>>>     content = fp.read()  # read all content
>>>     fp.seek(0)  # return to the start of the resource
>>>     lines = fp.readlines()  # read multiple lines
>>>     fp.write('Hello World')  # an error will be raised as resources cannot be written in read mode
>>>
>>> with o.open_resource('test_file_resource', mode='r+') as fp:  # enable read/write mode
>>>     fp.read()
>>>     fp.tell()  # current position
>>>     fp.seek(10)
>>>     fp.truncate()  # truncate the following content
>>>     fp.writelines(['Hello\n', 'World\n'])  # write multiple lines
>>>     fp.write('Hello World')
>>>     fp.flush()  # manual call submits the update to MaxCompute

The following open modes are supported:

  • r: Read mode. The file can be opened but cannot be written.
  • w: Write mode. The file can be written but cannot be read. Note that file content is cleared first if the file is opened in write mode.
  • a: Append mode. Content can be added to the end of the file.
  • r+: Read/write mode. You can read and write any content.
  • w+: Similar to r+, but file content is cleared first.
  • a+: Similar to r+, but content can be added to the end of the file only during writing.

In PyODPS, file resources can be opened in binary mode. For example, some compressed files must be opened in binary mode. rb indicates opening a file in binary read mode, and r+b indicates opening a file in binary read/write mode.

Table resources

Create a table resource

>>> o.create_resource('test_table_resource', 'table', table_name='my_table', partition='pt=test')

Update a table resource

>>> table_resource = o.get_resource('test_table_resource')
>>> table_resource.update(partition='pt=test2', project_name='my_project2')

Obtain associated table and partition

>>> table_resource = o.get_resource('test_table_resource')
>>> table = table_resource.table
>>> print(table.name)
>>> partition = table_resource.partition
>>> print(partition.spec)

Read and write table

>>> table_resource = o.get_resource('test_table_resource')
>>> with table_resource.open_writer() as writer:
>>>     writer.write([0, 'aaaa'])
>>>     writer.write([1, 'bbbbb'])
>>> with table_resource.open_reader() as reader:
>>>     for rec in reader:
>>>         print(rec)