Migrating Drupal 7 File Entities to Drupal 8 Media Entities

The Drupal 8.3.x branch is getting ready to introduce a new experimental media module. This will bring enhanced media handling in Drupal 8. The closest solution in Drupal 7 to handle media is the file entity module. Now is the time to discuss migrations from file entity in Drupal 7 to media entities in Drupal 8. For core, there is already an issue for this, but for contrib... there is no migration. So, I wrote one.

by
jibran.ijaz
/ 24 March 2017

The file_entity module adds a type field to an existing {file_managed} table which allows creating file bundles for each file type. There is already a Drupal 7 to Drupal 8 files migration in core. Managed files are not fieldable entities in Drupal 7 so the file migration in core doesn't handle the file fields but file_entity module allows to add fields to file bundles.

To add support for 'types' and make the migration fieldable I wrote a new migrate source plugin. It adds a type filter to the query to allow migrating specific files. Also, I made the plugin extend from the FieldableEntity source plugin in core. To import the fields for the file type, I override the prepareRow method in the plugin.

The final source plugin looks like this:

<?php // modules/custom/my_custom_module/src/Plugin/migrate/source/FileEntity.phpnamespace Drupal\my_custom_module\Plugin\migrate\source;use Drupal\Core\Database\Query\Condition;use Drupal\migrate\Row;use Drupal\migrate_drupal\Plugin\migrate\source\d7\FieldableEntity;/** * Drupal 7 file_entity source from database. * * @MigrateSource( * id = "file_entity", * source_provider = "file" * ) */class FileEntity extends FieldableEntity { /** * {@inheritdoc} */ public function query() { $query = $this->select('file_managed', 'f') ->fields('f') ->orderBy('f.fid'); if (isset($this->configuration['type'])) { $query->condition('f.type', $this->configuration['type']); } // Filter by scheme(s), if configured. if (isset($this->configuration['scheme'])) { $schemes = array(); // Accept either a single scheme, or a list. foreach ((array) $this->configuration['scheme'] as $scheme) { $schemes[] = rtrim($scheme) . '://'; } $schemes = array_map([$this->getDatabase(), 'escapeLike'], $schemes); // The uri LIKE 'public://%' OR uri LIKE 'private://%'. $conditions = new Condition('OR'); foreach ($schemes as $scheme) { $conditions->condition('uri', $scheme . '%', 'LIKE'); } $query->condition($conditions); } return $query; } /** * {@inheritdoc} */ public function prepareRow(Row $row) { // Get Field API field values. foreach (array_keys($this->getFields('file', $row->getSourceProperty('type'))) as $field) { $fid = $row->getSourceProperty('fid'); $row->setSourceProperty($field, $this->getFieldValues('file', $field, $fid)); } return parent::prepareRow($row); } /** * {@inheritdoc} */ public function fields() { return array( 'fid' => $this->t('File ID'), 'uid' => $this->t('The {users}.uid who added the file. If set to 0, this file was added by an anonymous user.'), 'filename' => $this->t('File name'), 'uri' => $this->t('The URI to access the file'), 'filemime' => $this->t('File MIME Type'), 'status' => $this->t('The published status of a file.'), 'timestamp' => $this->t('The time that the file was added.'), 'type' => $this->t('The type of this file.'), ); } /** * {@inheritdoc} */ public function getIds() { $ids['fid']['type'] = 'integer'; return $ids; }}

There were four file types in Drupal 7.

Audio
Image (local)
Videos (local)
Videos (youtube)

I installed four modules for this.

I also created four media bundles for these media type plugins.

Before writing the actual migrations, the file migration was needed.

# modules/custom/my_custom_module/migrations/my_files.ymlid: my_fileslabel: Filesmigration_tags: - Customsource: plugin: d7_file constants: source_base_path: 'sites/default/files/' old_files_path: 'sites/default/files/migration-files'process: filename: filename source_full_path: - plugin: concat delimiter: / source: - constants/old_files_path - filepath - plugin: urlencode uri: - plugin: skip_youtube_files source: - '@source_full_path' - uri - plugin: file_copy filemime: filemime # filesize is dynamically computed when file entities are saved, so there is # no point in migrating it. # filesize: filesize status: status created: timestamp changed: timestamp fid: fid uid: - plugin: skip_on_empty method: process source: uid - plugin: migration migration: my_usersdestination: plugin: entity:file migration_dependencies: required: - my_users

In Drupal 7, the media_youtube module is using Youtube stream wrapper to store the files. To ignore Youtube videos, we wrote a process plugin.

<?php # modules/custom/my_custom_module/src/Plugin/migrate/process/SkipYoutubeVideos.phpnamespace Drupal\my_custom_module\Plugin\migrate\process;use Drupal\migrate\MigrateExecutableInterface;use Drupal\migrate\MigrateSkipRowException;use Drupal\migrate\ProcessPluginBase;use Drupal\migrate\Row;/** * Skip youtube videos. * * @MigrateProcessPlugin( * id = "skip_youtube_files" * ) */class SkipYoutubeVideos extends ProcessPluginBase { /** * {@inheritdoc} */ public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) { if (parse_url(end($value), PHP_URL_SCHEME) == 'youtube') { throw new MigrateSkipRowException(); } return $value; }}

We kept the file id same between Drupal 7 and Drupal 8 so I decided to keep media entity id same as file id as well for the media migration. Each file type had a different set of fields so I wrote per bundle migrations.

To import audio files I configured the file type in migration also hard coded the target bundle to audio.

# modules/custom/my_custom_module/migrations/my_media_audio.ymlid: my_media_audiolabel: Media Audiomigration_tags: - Customsource: plugin: file_entity type: audio constants: bundle: 'audio'process: mid: fid bundle: 'constants/bundle' langcode: plugin: default_value source: language default_value: "und" name: filename uid: - plugin: skip_on_empty method: process source: uid - plugin: migration migration: my_users status: status created: timestamp changed: timestamp # File field see media_entity.bundle.audio.yml. field_media_audio/target_id: fid # Title field. field_title: field_title # Transcript field. field_transcript: field_transcriptdestination: plugin: entity:media migration_dependencies: required: - my_files - my_users

To migrate media images, I moved the dedicated field entity fields for alt and title to the image field.

# modules/custom/my_custom_module/migrations/my_media_audio.ymlid: my_media_imagelabel: Filesmigration_tags: - Customsource: plugin: file_entity type: image constants: bundle: 'image'process: mid: fid bundle: 'constants/bundle' langcode: plugin: default_value source: language default_value: "und" name: filename uid: - plugin: skip_on_empty method: process source: uid - plugin: migration migration: my_users status: status created: timestamp changed: timestamp # Image field see media_entity.bundle.image.yml. field_media_image/target_id: fid field_media_image/alt: field_file_image_alt_text/0/value field_media_image/title: field_file_image_title_text/0/value # Description field. field_description: field_image_description # Caption field. field_caption: field_captiondestination: plugin: entity:media migration_dependencies: required: - my_files - my_users

To import local video, I add the URI scheme to the video migration.

# modules/custom/my_custom_module/migrations/my_media_local_video.ymlid: my_media_local_videolabel: Filesmigration_tags: - Customsource: plugin: file_entity type: video # See output of SELECT DISTINCT(SUBSTRING_INDEX(uri, ':', 1)) FROM file_managed WHERE type = 'video'; scheme: - "public" constants: bundle: 'local_video'process: mid: fid bundle: 'constants/bundle' langcode: plugin: default_value source: language default_value: "und" name: filename uid: - plugin: skip_on_empty method: process source: uid - plugin: migration migration: my_users status: status created: timestamp changed: timestamp # File field see media_entity.bundle.local_video.yml. field_media_video/target_id: fid # Title field. field_title: field_video_title # Transcript field. field_transcript: field_transcriptdestination: plugin: entity:media migration_dependencies: required: - my_files - my_users

To import the youtube videos the URI was like youtube://v/video_id but embed field needs the youtube video URL so I created a process plugin to convert the video ID to URL.

# modules/custom/my_custom_module/migrations/my_media_video.ymlid: my_media_videolabel: Filesmigration_tags: - Customsource: plugin: file_entity type: video # See output of SELECT DISTINCT(SUBSTRING_INDEX(uri, ':', 1)) FROM file_managed WHERE type = 'video'; scheme: - "youtube" constants: bundle: 'video'process: mid: fid bundle: 'constants/bundle' langcode: plugin: default_value source: language default_value: "und" name: filename uid: - plugin: skip_on_empty method: process source: uid - plugin: migration migration: my_users status: status created: timestamp changed: timestamp # Embed field see media_entity.bundle.video.yml. field_media_video_embed_field: plugin: youtube source: uri # Title field. field_title: field_video_title # Transcript field. field_transcript: field_transcriptdestination: plugin: entity:media migration_dependencies: required: - my_files - my_users

The Youtube process plugin

<?php # modules/custom/my_custom_module/src/Plugin/migrate/process/Youtube.phpnamespace Drupal\my_module\Plugin\migrate\process;use Drupal\migrate\MigrateExecutableInterface;use Drupal\migrate\ProcessPluginBase;use Drupal\migrate\Row;/** * Custom process plugin to convert youtube scheme uri to video url. * * @MigrateProcessPlugin( * id = "youtube" * ) */class Youtube extends ProcessPluginBase { const SCHEME = 'youtube://'; const BASE_URL = 'http://youtube.com/watch?'; /** * {@inheritdoc} */ public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) { // Convert youtube scheme uri to video url. if (strpos($value, static::SCHEME) !== FALSE) { $value = static::BASE_URL . implode('=', explode('/', str_replace(static::SCHEME, '', $value), 2)); } else { $value = NULL; } return $value; }}

I ran these migrations using migrate_drush.

Conclusion

On paper, file entity to media migration sounds difficult, but the migration API in Drupal core and its use of new entity and plugin APIs made the migration for a custom entity with complex fields a lot easier.

Thanks, to Lee Rowlands for writing the skip_youtube_files plugin. Also, thank you, Ben Dougherty, for all the advice.

The code can also be found as a gist.

Tagged

Drupal Development, Drupal Site Building

Posted by
jibran.ijaz
Drupal Developer

Dated 24 March 2017

Comments

Comment by
Lucas Hedding

Dated 31 March 2017

Also very possible is to convert your entire file ecosystem into Media items by adding in extra ER relationships instead of the direct file relationship. Both when converting from d6 or d7 to D8.

Comment by
jibran.ijaz

Dated 31 March 2017

Yes, but some files can't be treated as media for example compressed files.

Pagination

Add new comment

Original Article:

Migrating Drupal 7 File Entities to Drupal 8 Media Entities