• Object Storage Service

obs
  1. Help Center
  2. Object Storage Service
  3. Developer Guide (Node.js SDK)
  4. Object Upload
  5. Performing a Multipart Upload

Performing a Multipart Upload

To upload a large file, multipart upload is recommended. Multipart upload is applicable to many scenarios, including:

  • Files to be uploaded are larger than 100 MB.
  • The network condition is poor. Connection to the OBS server is constantly down.
  • Sizes of files to be uploaded are uncertain.

Multipart upload consists of three phases:

  1. Initialize a multipart upload (ObsClient.initiateMultipartUpload).
  2. Upload parts one by one or concurrently (ObsClient.uploadPart).
  3. Combine parts (ObsClient.completeMultipartUpload) or abort the multipart upload (ObsClient.abortMultipartUpload).

Initializing a Multipart Upload

Before upload, you need to notify OBS of initializing a multipart upload. This operation will return an upload ID (globally unique identifier) created by the OBS server to identify the multipart upload. You can use this upload ID to initiate related operations, such as aborting the multipart upload, listing multipart uploads, and listing uploaded parts.

You can call ObsClient.initiateMultipartUpload to initialize a multipart upload.

// Import the OBS library.
var ObsClient = require('./lib/obs');

// Create an instance of ObsClient.
var obsClient = new ObsClient({
    access_key_id: '*** Provide your Access Key ***',       
    secret_access_key: '*** Provide your Secret Key ***',       
    server : 'yourdomainname'
});

obsClient.initiateMultipartUpload({
       Bucket : 'bucketname',
       Key : 'objectkey',
       ContentType : 'text/plain',
       Metadata : {'property' : 'property-value'}
}, (err, result) => {
       if(err){
              console.error('Error-->' + err);
       }else{
              console.log('Status-->' + result.CommonMsg.Status);
              if(result.CommonMsg.Status < 300 && result.InterfaceResult){
                     console.log('UploadId-->' + result.InterfaceResult.UploadId);
              }
       }
});
NOTE:
  • When initializing a multipart upload, you can use the ContentType and Metadata parameters to respectively set the MIME type and customize the metadata of an object, besides the object name and owning bucket.
  • After the API for initializing a multipart upload is called, the upload ID will be returned. This ID will be used in follow-up operations.

Uploading a Part

After initializing a multipart upload, you can specify the object name and upload ID to upload a part. Each upload part has a part number (ranging from 1 to 10000). For parts with the same upload ID, their part numbers are unique and identify their comparative locations in the object. If you use the same part number to upload two parts, the latter one being uploaded will overwrite the former. Except for the part last uploaded whose size ranges from 0 to 5 GB, sizes of the other parts range from 100 KB to 5 GB. Parts are uploaded in random order and can be uploaded through different processes or machines. OBS will combine them into the object based on their part numbers.

You can call ObsClient.uploadPart to upload a part.

// Import the OBS library.
var ObsClient = require('./lib/obs');

// Create an instance of ObsClient.
var obsClient = new ObsClient({
    access_key_id: '*** Provide your Access Key ***',       
    secret_access_key: '*** Provide your Secret Key ***',       
    server : 'yourdomainname'
});

obsClient.uploadPart({
       Bucket:'bucketname',
       Key:'objectkey',
       // Set the part number, which ranges from 1 to 10000.
       PartNumber:1,
       // Set the upload ID.
       UploadId:'upload id from initiateMultipartUpload',
      // Set the large file to be uploaded.
       SourceFile: 'localfile',
       // Set the part size.
       PartSize: 5 * 1024 * 1024,
       // Set the start offset.
       Offset: 0
}, (err, result) => {
       if(err){
              console.log('Error-->' + err);
       }else{
              console.log('Status-->' + result.CommonMsg.Status);
              if(result.CommonMsg.Status < 300 && result.InterfaceResult){
                     console.log('ETag-->' + result.InterfaceResult.ETag);
              }
       }
});
NOTE:
  • Use the PartNumber parameter to specify the part number, the UploadId parameter to specify the globally unique ID, the SourceFile parameter to specify the to-be-uploaded file, the PartSize parameter to set the part size, and the Offset parameter to set the start offset of a part.
  • Except the part last uploaded, other parts must be larger than 100 KB. Part sizes will not be verified during upload because which one is last uploaded is not identified until parts are combined.
  • OBS will return ETags (MD5 values) of the received parts to users.
  • You can use the ContentMD5 parameter to set the MD5 value for the object to be uploaded, which can be provided to the OBS server for data integrity verification.
  • Part numbers range from 1 to 10000. If a part number exceeds this range, OBS will return a 400 Bad Request error.

Combining Parts

After all parts are uploaded, call the API for combining parts to generate the object. Before this operation, valid part numbers and ETags of all parts must be sent to OBS. After receiving this information, OBS verifies the validity of each part one by one. After all parts pass the verification, OBS combines these parts to form the final object.

You can call ObsClient.completeMultipartUpload to combine parts.

// Import the OBS library.
var ObsClient = require('./lib/obs');

// Create an instance of ObsClient.
var obsClient = new ObsClient({
    access_key_id: '*** Provide your Access Key ***',       
    secret_access_key: '*** Provide your Secret Key ***',       
    server : 'yourdomainname'
});

obsClient.completeMultipartUpload({
       Bucket:'bucketname',
       Key:'objectkey',
       // Set the upload ID.
       UploadId:'upload id from initiateMultipartUpload',
       Parts: [{'PartNumber':1,'ETag':'etag value from uploadPart'}]
}, (err, result) => {
       if(err){
              console.log('Error-->' + err);
       }else{
              console.log('Status-->' + result.CommonMsg.Status);
       }
});
NOTE:
  • Use the UploadId parameter to specify the upload ID for the multipart upload and the Parts parameter to specify the list of part numbers and ETags. Content in the list is displayed in the ascending order by part number.
  • Part numbers can be inconsecutive.

Concurrently Uploading Parts

Multipart upload is mainly used for large file upload or when the network condition is poor. The following sample code shows how to concurrently upload large files in multipart mode:

// Import the OBS library.
var ObsClient = require('./lib/obs');

// Create an instance of ObsClient.
var obsClient = new ObsClient({
    access_key_id: '*** Provide your Access Key ***',       
    secret_access_key: '*** Provide your Secret Key ***',       
    server : 'yourdomainname'
});

// Initialize a multipart upload.
obsClient.initiateMultipartUpload({
       Bucket : 'bucketname',
       Key : 'objectkey'
}, (err, result) => {
       if(!err && result.CommonMsg.Status < 300){
              var uploadId = result.InterfaceResult.UploadId;
              console.log('\t' + uploadId + '\n');
              // Set the part size to 100 MB.
              var partSize = 100 * 1024 * 1024;
              var fs = require('fs');
              var sampleFilePath = 'localfile';
              var fileLength = fs.lstatSync(sampleFilePath).size;
// Calculate the number of parts to be uploaded.
              var partCount = fileLength % partSize === 0 ? Math.floor(fileLength / partSize) :  Math.floor(fileLength / partSize) + 1;
              var events = require('events');
              var eventEmitter = new events.EventEmitter();
              var parts = [];
              // Specify that the parts will be uploaded concurrently.
              for(let i=0;i<partCount;i++){
                     // Set the start position of a part in the file.
                     let offset = i * partSize;
                     // Part size
                     let currPartSize = (i + 1 === partCount) ? fileLength - offset : partSize;
                     // Part number
                     let partNumber = i + 1;
                     obsClient.uploadPart({
                           Bucket : 'bucketname',
                           Key: 'objectkey',
                           PartNumber: partNumber,
                           UploadId : uploadId,
                           SourceFile: sampleFilePath,
                           Offset : offset,
                           PartSize : currPartSize
                     }, (err, result) => {
                           if(!err && result.CommonMsg.Status < 300){
                                  parts.push({PartNumber : partNumber, ETag : result.InterfaceResult.ETag});
                                  if(parts.length === partCount){
                                         // Set the part numbers and ETags to be listed in ascending order by part number.
                                         var _parts = parts.sort((a, b) => {
                                                if(a.PartNumber >= b.PartNumber){
                                                       return 1;
                                                }
                                                return -1;
                                         });
                                         eventEmitter.emit('upload part finished', _parts);
                                  }
                           }else{
                                  throw new Error(err || result.CommonMsg.Code);
                           }
                     });
              }
              // Wait until the upload is complete.
              eventEmitter.on('upload part finished', (parts) => {
                     // Combine parts.
                     obsClient.completeMultipartUpload({
                           Bucket : 'bucketname',
                           Key: 'objectkey',
                           UploadId: uploadId,
                           Parts: parts
                     }, (err, result) => {
                           if(err){
                                  console.log('Error-->' + err);
                           }else{
                                  console.log('Status-->' + result.CommonMsg.Status);
                           }
                     });
              });
       }
});
NOTE:

When uploading a large file in multipart mode, you need to use the Offset and PartSize parameters to set the start and end positions of each part in the file.

Aborting a Multipart Upload

After a multipart upload is aborted, you cannot use its upload ID to perform any operation and the uploaded parts will be deleted by OBS.

You can call ObsClient.abortMultipartUpload to abort a multipart upload.

// Import the OBS library.
var ObsClient = require('./lib/obs');

// Create an instance of ObsClient.
var obsClient = new ObsClient({
    access_key_id: '*** Provide your Access Key ***',       
    secret_access_key: '*** Provide your Secret Key ***',       
    server : 'yourdomainname'
});

obsClient.abortMultipartUpload({
       Bucket:'bucketname',
       Key:'objectkey',
       // Set the upload ID.
       UploadId:'upload id from initiateMultipartUpload',
}, (err, result) => {
       if(err){
              console.log('Error-->' + err);
       }else{
              console.log('Status-->' + result.CommonMsg.Status);
       }
});

Listing Uploaded Parts

You can call ObsClient.listParts to list successfully uploaded parts of a multipart upload.

The following table describes the parameters involved in this API.

Parameter

Description

UploadId

Upload ID, which globally identifies a multipart upload. The value is in the returned result of ObsClient.initiateMultipartUpload.

MaxParts

Maximum number of parts that can be listed per page.

PartNumberMarker

Part number after which listing uploaded parts begins. Only parts whose part numbers are larger than this value will be listed.

  • Listing parts in simple mode
// Import the OBS library.
var ObsClient = require('./lib/obs');

// Create an instance of ObsClient.
var obsClient = new ObsClient({
    access_key_id: '*** Provide your Access Key ***',       
    secret_access_key: '*** Provide your Secret Key ***',       
    server : 'yourdomainname'
});

// List uploaded parts. uploadId is obtained from initiateMultipartUpload. 
obsClient.listParts({
       Bucket : 'bucketname',
       Key: 'objectkey',
       UploadId : 'upload id from initiateMultipartUpload'
}, (err, result) => {
       if(err){
              console.log('Error-->' + err);
       }else{
              console.log('Status-->' + result.CommonMsg.Status);
              if(result.CommonMsg.Status < 300 && result.InterfaceResult){
                     for(let i=0;i<result.InterfaceResult.Parts.length;i++){
                           console.log('Part['+ i +']:');
                           // Part number, specified during the upload
                           console.log('PartNumber-->' + result.InterfaceResult.Parts[i]['PartNumber']);
                           // Time when the part was last uploaded
                           console.log('LastModified-->' + result.InterfaceResult.Parts[i]['LastModified']);
                           // Part ETag
                           console.log('ETag-->' + result.InterfaceResult.Parts[i]['ETag']);
                           // Part size
                           console.log('Size-->' + result.InterfaceResult.Parts[i]['Size']);
                     }
              }
       }
});
NOTE:
  • A maximum of 1000 parts can be returned each time. If an upload of the specific upload ID contains more than 1000 parts and InterfaceResult.IsTruncated is true in the returned result, not all parts are returned. In such cases, you can use InterfaceResult.NextPartNumberMarker to obtain the start position for next listing.
  • If you want to obtain all parts involved in a specific upload ID, you can use the paging mode for listing.
  • Listing all parts

If the number of parts of a multipart upload is larger than 1000, you can use the following sample code to list all parts.

// Import the OBS library.
var ObsClient = require('./lib/obs');

// Create an instance of ObsClient.
var obsClient = new ObsClient({
    access_key_id: '*** Provide your Access Key ***',       
    secret_access_key: '*** Provide your Secret Key ***',       
    server : 'yourdomainname'
});

var listAll = (partNumberMarker) => {
       // List uploaded parts. uploadId is obtained from initiateMultipartUpload. 
       obsClient.listParts({
              Bucket : 'bucketname',
              Key: 'objectkey',
              UploadId : 'upload id from initiateMultipartUpload',
              PartNumberMarker : partNumberMarker
       }, (err, result) => {
              if(err){
                     console.log('Error-->' + err);
              }else{
                     console.log('Status-->' + result.CommonMsg.Status);
                     if(result.CommonMsg.Status < 300 && result.InterfaceResult){
                           for(let i=0;i<result.InterfaceResult.Parts.length;i++){
                                  console.log('Part['+ i +']:');
                                  // Part number, specified during the upload
                                  console.log('PartNumber-->' + result.InterfaceResult.Parts[i]['PartNumber']);
                                  // Time when the part was last uploaded
                                  console.log('LastModified-->' + result.InterfaceResult.Parts[i]['LastModified']);
                                  // Part ETag
                                  console.log('ETag-->' + result.InterfaceResult.Parts[i]['ETag']);
                                  // Part size
                                  console.log('Size-->' + result.InterfaceResult.Parts[i]['Size']);
                           }
                           if(result.InterfaceResult.IsTruncated === 'true'){
                                  listAll(result.InterfaceResult.NextPartNumberMarker);
                           }
                     }
              }
       });
};

listAll();

  • Listing all parts in paging mode

The previously described listing (1000 parts per page) is a special paging listing mode. The following sample code shows how to specify the number of parts displayed per page when listing.

// Import the OBS library.
var ObsClient = require('./lib/obs');

// Create an instance of ObsClient.
var obsClient = new ObsClient({
    access_key_id: '*** Provide your Access Key ***',       
    secret_access_key: '*** Provide your Secret Key ***',       
    server : 'yourdomainname'
});

var listAll = (partNumberMarker) => {
       // List uploaded parts. uploadId is obtained from initiateMultipartUpload. 
       obsClient.listParts({
              Bucket : 'bucketname',
              Key: 'objectkey',
              UploadId : 'upload id from initiateMultipartUpload',
              // Set the number of parts displayed per page to 100.
              MaxParts : 100,
              PartNumberMarker : partNumberMarker
       }, (err, result) => {
              if(err){
                     console.log('Error-->' + err);
              }else{
                     console.log('Status-->' + result.CommonMsg.Status);
                     if(result.CommonMsg.Status < 300 && result.InterfaceResult){
                           for(let i=0;i<result.InterfaceResult.Parts.length;i++){
                                  console.log('Part['+ i +']:');
                                  // Part number, specified during the upload
                                  console.log('PartNumber-->' + result.InterfaceResult.Parts[i]['PartNumber']);
                                  // Time when the part was last uploaded
                                  console.log('LastModified-->' + result.InterfaceResult.Parts[i]['LastModified']);
                                 // Part ETag
                                  console.log('ETag-->' + result.InterfaceResult.Parts[i]['ETag']);
                                  // Part size
                                  console.log('Size-->' + result.InterfaceResult.Parts[i]['Size']);
                           }
                           if(result.InterfaceResult.IsTruncated === 'true'){
                                  listAll(result.InterfaceResult.NextPartNumberMarker);
                           }
                     }
              }
       });
};

listAll();

Listing Multipart Uploads

You can call ObsClient.listMultipartUploads to list multipart uploads. The following table describes parameters involved in ObsClient.listMultipartUploads.

Parameter

Description

Prefix

Prefix that the object names in the multipart uploads to be listed must contain

Delimiter

Character used to group object names involved in multipart uploads. If the object name contains the Delimiter parameter, the character string from the first character to the first Delimiter parameter in the object name is grouped under a single result element, CommonPrefix. (If a prefix is specified in the request, the prefix must be removed from the object name.)

MaxUploads

Maximum number of returned multipart uploads. The value ranges from 1 to 1000. If the value is not in this range, 1000 uploads are returned by default.

KeyMarker

Object name to start with when listing multipart uploads

UploadIdMarker

Upload ID after which the multipart upload listing begins. It is effective only when used with KeyMarker so that multipart uploads after UploadIdMarker of KeyMarker will be listed.

  • Listing multipart uploads in simple mode
// Import the OBS library.
var ObsClient = require('./lib/obs');

// Create an instance of ObsClient.
var obsClient = new ObsClient({
    access_key_id: '*** Provide your Access Key ***',       
    secret_access_key: '*** Provide your Secret Key ***',       
    server : 'yourdomainname'
});

obsClient.listMultipartUploads({
       Bucket : 'bucketname'
}, (err, result) => {
       if(err){
              console.log('Error-->' + err);
       }else{
              console.log('Status-->' + result.CommonMsg.Status);
              if(result.CommonMsg.Status < 300 && result.InterfaceResult){
                     for(let i=0;i<result.InterfaceResult.Uploads.length;i++){
                           console.log('Uploads[' + i + ']');
                           console.log('UploadId-->' + result.InterfaceResult.Uploads[i]['UploadId']);
                           console.log('Key-->' + result.InterfaceResult.Uploads[i]['Key']);
                           console.log('Initiated-->' + result.InterfaceResult.Uploads[i]['Initiated']);
                     }
              }
       }
});
NOTE:
  • Information about a maximum of 1000 multipart uploads can be returned each time. If a bucket contains more than 1000 multipart uploads and InterfaceResult.IsTruncated is true in the returned result, not all uploads will be returned. In such cases, you can use InterfaceResult.NextKeyMarker and InterfaceResult.NextUploadIdMarker to obtain the start position for next listing.
  • If you want to obtain all multipart uploads in a bucket, you can use the paging listing.
  • Listing all multipart uploads
// Import the OBS library.
var ObsClient = require('./lib/obs');

// Create an instance of ObsClient.
var obsClient = new ObsClient({
    access_key_id: '*** Provide your Access Key ***',       
    secret_access_key: '*** Provide your Secret Key ***',       
    server : 'yourdomainname'
});

var listAll = (keyMarker, uploadIdMarker) => {
       obsClient.listMultipartUploads({
              Bucket : 'bucketname',
              KeyMarker : keyMarker,
              UploadIdMarker : uploadIdMarker
       }, (err, result) => {
              if(err){
                     console.log('Error-->' + err);
              }else{
                     console.log('Status-->' + result.CommonMsg.Status);
                     if(result.CommonMsg.Status < 300 && result.InterfaceResult){
                           for(let i=0;i<result.InterfaceResult.Uploads.length;i++){
                                  console.log('Uploads[' + i + ']');
                                  console.log('UploadId-->' + result.InterfaceResult.Uploads[i]['UploadId']);
                                  console.log('Key-->' + result.InterfaceResult.Uploads[i]['Key']);
                                  console.log('Initiated-->' + result.InterfaceResult.Uploads[i]['Initiated']);
                           }
                           if(result.InterfaceResult.IsTruncated === 'true'){
                                  listAll(result.InterfaceResult.NextKeyMarker, result.InterfaceResult.NextUploadIdMarker);
                           }
                     }
              }
       });
}

listAll();
  • Listing all multipart uploads in paging mode

The previous sample code (listing 1000 uploads per page) is a special paging listing mode. The following sample code shows how to specify the number of uploads displayed per page when listing.

// Import the OBS library.
var ObsClient = require('./lib/obs');

// Create an instance of ObsClient.
var obsClient = new ObsClient({
    access_key_id: '*** Provide your Access Key ***',       
    secret_access_key: '*** Provide your Secret Key ***',       
    server : 'yourdomainname'
});

var listAll = (keyMarker, uploadIdMarker) => {
       obsClient.listMultipartUploads({
              Bucket : 'bucketname',
              KeyMarker : keyMarker,
              UploadIdMarker : uploadIdMarker,
              // Set the number of parts displayed per page to 100.
              MaxUploads : 100
       }, (err, result) => {
              if(err){
                     console.log('Error-->' + err);
              }else{
                     console.log('Status-->' + result.CommonMsg.Status);
                     if(result.CommonMsg.Status < 300 && result.InterfaceResult){
                           for(let i=0;i<result.InterfaceResult.Uploads.length;i++){
                                  console.log('Uploads[' + i + ']');
                                  console.log('UploadId-->' + result.InterfaceResult.Uploads[i]['UploadId']);
                                  console.log('Key-->' + result.InterfaceResult.Uploads[i]['Key']);
                                  console.log('Initiated-->' + result.InterfaceResult.Uploads[i]['Initiated']);
                           }
                           if(result.InterfaceResult.IsTruncated === 'true'){
                                  listAll(result.InterfaceResult.NextKeyMarker, result.InterfaceResult.NextUploadIdMarker);
                           }
                     }
              }
       });
}

listAll();