Efficient ETL development with XDTL

XDTL is an XML based descriptional language designed by Mindworks Industries for specifying data transformations from one database/storage to another. XDTL/XDTL Runtime Engine is built not from the perspective of a slick drag-and-drop IDE, but an efficient language for describing data transformations. The goal is to produce a lightweight ETL development/runtime environment that would handle most of the common requirements with better efficiency than traditional jack-of-all-trades tools. XDTL takes advantage of the general-purpose metadata repository of MMX Framework targeting a broad range of metadata-related activities and not locking the user into an ETL-specific and ETL-only repository from one vendor.

As data amounts grow, the paradigm shifts from ETL to ELT, where bulk of the transformations take place inside the (target) database. Most of the fancy features provided by heavyweight ETL tools are rarely or never used and the main workforce is SQL. Based on our experience (and tons of ETL code written) we have extracted a set of common SQL "patterns" and converted them into templates for processing by a template engine. Template engine merges templates and mappings stored in the repository to create executable SQL statement "instances".

More information available at www.xdtl.org