dataset-evaluation

Solid

Validates dataset formatting and quality for SageMaker model fine-tuning (SFT, DPO, or RLVR). Use when the user says "is my dataset okay", "evaluate my data", "check my training data", "I have my own data", or before starting any fine-tuning job. Detects file format, checks schema compliance against the selected model and technique, and reports whether the data is ready for training or evaluation.

Data & Documents 765 stars 108 forks Updated 2 days ago Apache-2.0

Install

View on GitHub

Quality Score: 95/100

Stars 20%
96
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Workflow Instruction Follow the workflow shown below. Locate the dataset, check the file type, and resolve any issues with missing files or wrong file types. Determine the fine-tuning model and fine-tuning strategy. Run scripts/format_detector.py to evaluate whether the file is formatted correctly for the currently selected model and strategy. Summarize the results: is the dataset ready for fine-tuning? ## Workflow 1. **Locate Dataset**: - The full path may be a local file path, or an S3 URI - Resolve the full path to the dataset file, make sure read permissions are available, and help the user if the file is not found 2. **Determine strategy and model**: - File formatting depends on the currently selected fine-tuning strategy and fine-tuning base model. - If the strategy and model are already known from the conversation context (e.g., selected via the finetuning-setup skill), use them. - If not available in context, activate the finetuning-setup skill to determine them before proceeding. 3. **Check File Formatting**: Run the tool format_detector.py to make sure the file conforms to formatting requirements. - Send the full path directly to the format_detector script as an argument - Do not send the model and strategy as arguments - Do not download data from S3 - Do not make local copies of data 4. **Summarize Results**: Tell the user if their data is ready - Examine the output of format_detector and compare to the known strategy and model...

Details

Author
awslabs
Repository
awslabs/agent-plugins
Created
3 months ago
Last Updated
2 days ago
Language
Shell
License
Apache-2.0

Similar Skills

Semantically similar based on skill content — not just same category

Data & Documents Listed

dataset-curator

Use this skill when designing, cleaning, deduplicating, or documenting datasets for model training and evaluation including schema design, class imbalance handling, and train/val/test splits. Not for running model training or hyperparameter tuning. Not for real-time data pipeline engineering.

15 Updated 2 days ago
NickCrew
Data & Documents Solid

dataset-transformation

Generates a Jupyter notebook that transforms datasets between ML schemas for model training or evaluation. Use when the user says "transform", "convert", "reformat", "change the format", or when a dataset's schema needs to change to match the target format — always use this skill for format changes rather than writing inline transformation code. Supports OpenAI chat, SageMaker SFT/DPO/RLVR, HuggingFace preference, Bedrock Nova, VERL, and custom JSONL formats from local files or S3.

765 Updated 2 days ago
awslabs
AI & Automation Solid

model-evaluation

Generates a Jupyter notebook that evaluates a fine-tuned SageMaker model using LLM-as-a-Judge. Use when the user says "evaluate my model", "how did my model perform", "compare models", or after a training job completes. Supports built-in and custom evaluation metrics, evaluation dataset setup, and judge model selection.

765 Updated 2 days ago
awslabs
AI & Automation Solid

finetuning

Generates a Jupyter notebook that fine-tunes a base model using SageMaker serverless training jobs. Use when the user says "start training", "fine-tune my model", "I'm ready to train", or when the plan reaches the finetuning step. Supports SFT, DPO, and RLVR trainers, including RLVR Lambda reward function creation.

765 Updated 2 days ago
awslabs
Data & Documents Listed

datarobot-data-preparation

Tools and guidance for data upload, dataset management, data validation, and preparing data for DataRobot projects. Use when uploading datasets, managing data, or validating data for DataRobot.

16 Updated 2 days ago
datarobot-oss