We lifted 9 functions out of this paper's own repositories and ran 6 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| mkuchnik/plumbertensorflow | canonical | 6 of 9 |
| Function | Status | Where it lives |
|---|---|---|
| cygpath | Ran | mkuchnik/plumbertensorflow/configure.py code served (permissive licence) · get_code("50ac80dd2372bb51") |
| get_input | Ran | mkuchnik/plumbertensorflow/configure.py code served (permissive licence) · get_code("e07e303581bd4892") |
| get_inputs_outputs | Ran | mkuchnik/plumbertensorflow/tensorflow/lite/python/convert_saved_model.py code served (permissive licence) · get_code("0031aaf5d0b6eef0") |
| minimize_loss_example | Ran | mkuchnik/plumbertensorflow/tensorflow/python/distribute/single_loss_example.py code served (permissive licence) · get_code("2034af57f4bcd949") |
| run_shell | Ran | mkuchnik/plumbertensorflow/configure.py code served (permissive licence) · get_code("8ebf21a48f9d476d") |
| single_loss_example | Ran | mkuchnik/plumbertensorflow/tensorflow/python/distribute/single_loss_example.py code served (permissive licence) · get_code("ce79211ba810feb6") |
| batchnorm_example | Not yet run | mkuchnik/plumbertensorflow/tensorflow/python/distribute/single_loss_example.py code served (permissive licence) · get_code("2cd4fe48e03992fc") |
| get_meta_graph_def | Not yet run | mkuchnik/plumbertensorflow/tensorflow/lite/python/convert_saved_model.py code served (permissive licence) · get_code("9a832e63833396e4") |
| get_signature_def | Not yet run | mkuchnik/plumbertensorflow/tensorflow/lite/python/convert_saved_model.py code served (permissive licence) · get_code("1a24b33d13309a22") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Input pipelines, which ingest and transform input data, are an essential part of training Machine Learning (ML) models. However, it is challenging to implement efficient input pipelines, as it requires reasoning about parallelism, asynchrony, and variability in fine-grained profiling information. Our analysis of over two million ML jobs in Google datacenters reveals that a significant fraction of model training jobs could benefit from faster input data pipelines. At the same time, our analysis indicates that most jobs do not saturate host hardware, pointing in the direction of software-based bottlenecks. Motivated by these findings, we propose Plumber, a tool for finding bottlenecks in ML input pipelines. Plumber uses an extensible and interpretable operational analysis analytical model to automatically tune parallelism, prefetching, and caching under host resource constraints. Across five representative ML pipelines, Plumber obtains speedups of up to 47x for misconfigured pipelines. By automating caching, Plumber obtains end-to-end speedups of over 50% compared to state-of-the-art tuners.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2111.04131")
get_code_for_paper("2111.04131")
have("2111.04131")
Connect an agent — have() is free.