As sustainability turns into extra vital to the general public, firms publish experiences to showcase their environmental and social targets. Nevertheless, not all claims are grounded in proof — some are deceptive, a observe referred to as greenwashing.
To assist detect such claims, we constructed an software referred to as GoalSpotter: a modular system that detects and analyzes sustainability aims in firm experiences utilizing 4 Cog containers, every answerable for a definite step within the pipeline :
🧠 Aim Detection → 🔍 Matter Detection → 📋 Element Extraction → ⏳ Time Extraction
The fashions used on this system have been developed by Tom Debus and @Mohammad Mahdavi . Test github repo for more information: https://github.com/Ferris-Options/goalspotter_public . My position was to containerize the fashions, orchestrate the pipeline, and construct the frontend as a Mesop app. On this article, I’ll stroll by means of how I constructed and related the containers to create the total pipeline.
⚙️ What’s Cog?
Cog is an open-source software that packages machine studying fashions into production-ready Docker containers. Meaning when you’ve acquired your mannequin working domestically, it’s simple to run it in manufacturing. Every container simply wants a cog.yaml for the setup and a predict.py with the prediction logic.
To put in Cog on macOS, simply run:
brew set up cog
For our app, we created 4 Cog containers:
goal-detectiontopic-detectiondetail-extractiontime-extraction
These containers run sequentially to course of and analyze sustainability content material in firm experiences.
🧭Step 1: Aim Detection Container
Aim Detection container kickstarts the pipeline. It takes an organization report (PDF or URL), breaks it into textual content blocks, and classifies whether or not every block comprises a sustainability purpose.
What Occurs Inside?
- Enter: Person uploads a PDF or submits a URL
- Textual content Extraction: The doc is parsed and segmented into particular person textual content blocks.
- Preprocessing: Blocks are cleaned and filtered to take away noise.
- Prediction: A pre-trained transformer mannequin classifies every block as “Aim” or “Not Aim” and assigns a confidence rating.
- Output: The outcomes are sorted by confidence and saved to a brief CSV file.
🧠 Key Snippet (from predict.py):
This masses the fine-tuned transformer mannequin used for purpose classification:
def setup(self):
self.machine = "cuda" if torch.cuda.is_available() else "cpu"
self.target_values = ["Not Goal", "Goal"]
self.goal_detection_model = transformer_model.TextClassification(
self.target_values, identify="distilroberta-base", load_from="goal-detection"
)
Detects whether or not the enter is a URL or a file:
if input_source.startswith(("http://", "https://")):
is_url = True
content_type = "html"
supply = input_source
elif input_source.startswith("knowledge:software/pdf;base64,"):
is_url = False
content_type = "pdf"
supply = file_path
else:
increase ValueError("Invalid enter supply")
Parses the content material, segments textual content, and cleans it:
parsed_content = doc.parse_content(content material)
text_blocks = doc.segment_text(parsed_content)
sentences = doc.get_sentences(text_blocks)tdf = pd.DataFrame({"Supply": supply, "Textual content Blocks": sentences})
tdf["text"] = tdf["Text Blocks"].copy()
tdf = data_preprocessor.clean_text_blocks(tdf, "textual content", stage="important")
tdf = data_preprocessor.filter_text_blocks(tdf, "textual content", keep_only_size=(0, 300))
Runs the classification mannequin and provides the prediction scores to the DataFrame:
predictions = self.goal_detection_model.predict(tdf["text"].tolist())
tdf["Goal Score"] = predictions["Goal"].values
tdf = tdf.drop(["text"], axis=1).sort_values("Aim Rating", ascending=False)
Shops the ends in a brief file that Cog can return:
temp_dir = tempfile.mkdtemp()
temp_output_path = Path(temp_dir) / "goal_detection.csv"
tdf.to_csv(temp_output_path, index=False)
return Path(temp_output_path)
📄 Wrapping it in cog.yaml
To inform Cog easy methods to construct and run your mannequin, we have to outline a cog.yaml file. This consists of specifying Python dependencies and elective GPU help.
Right here’s a snippet from the cog.yaml config:
construct:
gpu: true # Set to true should you plan to make use of GPU
python_version: "3.10"
python_requirements: necessities.txt
run:
- "python -m spacy obtain en_core_web_sm"
predict: "predict.py:Predictor"
💡 Be aware for M1 Mac customers: Putting in
torchmight trigger points. In yournecessities.txt, use the next line to make sure compatibility:
torch==2.0.1 --extra-index-url https://obtain.pytorch.org/whl/cpu
