從深度學習轉向統計模型後，我開始質疑整個機器學習項目

Hi everyone!

I'm currently working on a 'machine learning project', and honestly, I think I've reached the point where I'm questioning almost every technical decision I've made.

I could really use some advice from people who have been through something similar.

How it started

When I first started planning this project, I was interested in using deep learning or Explainable AI (XAI).

I wanted to build something meaningful, technically interesting, and ideally capable of producing insights that people could actually understand.

The project eventually evolved into a forecasting system for operational/business data, with an additional LLM component intended to translate forecasting results into business-friendly narratives.

It sounded reasonable in my head.

But implementing it has been a completely different experience.

How it's going

Somewhere along the way, I moved away from my original deep learning/XAI ideas and started experimenting with simpler statistical forecasting approaches.

And now I'm wondering:

Did I simplify the problem because it genuinely made more sense?

Or did I simplify it because I didn't fully understand how to approach the more complex methods?

I honestly don't know yet.

The more I learn about forecasting, the more I realize that choosing a model isn't just about picking the most advanced algorithm.

There are questions about data characteristics, assumptions, evaluation, temporal validation, and whether a model is even appropriate for the problem being solved.

I'm starting to realize how much I still need to learn.

What I've built so far

At the moment, my experimental pipeline roughly follows this structure:

Historical Data → Statistical Forecasting → Backtesting → Evaluation → LLM-Generated Business Narrative

I'm using Python and Jupyter Notebook, currently working with dummy/synthetic data.

I've also experimented with Gemma 2B for the narrative-generation component, although I ran into GPU memory limitations during inference.

The system is still experimental, and I haven't established whether my current approach is reliable enough for real-world use.

Where I'm genuinely stuck

1. Model selection

How do you actually decide whether a forecasting problem needs deep learning, traditional machine learning, or a statistical model?

I know that model complexity doesn't automatically translate into better performance.

But what criteria should guide that decision?

2. Explainable AI

I originally wanted to explore XAI, but I'm no longer sure how it fits into my current project.

If I'm using relatively interpretable statistical models, would adding XAI provide meaningful value?

Or am I trying to force a technique into a problem that doesn't necessarily need it?

3. Forecasting evaluation

I'm learning about backtesting and forecasting metrics, but I'm not fully confident in how to evaluate the system correctly.

How can I determine whether the model is genuinely learning useful patterns rather than just producing plausible-looking predictions?

What should I be checking beyond standard error metrics?

4. LLM integration

I'm also questioning my decision to include an LLM.

My original intention was to generate understandable business narratives from forecasting outputs.

But would a deterministic, rule-based approach be more appropriate?

How do people ensure that LLM-generated explanations remain faithful to the actual numerical results?

And is it reasonable to invest in this component before the forecasting methodology itself is solid?

5. Project architecture

Most of my work currently lives in a Jupyter Notebook.

I'm unsure how to move from experimentation toward a properly structured, reproducible project.

What should I prioritize first: improving the methodology, refactoring the code, establishing tests, or designing the integration with the larger application?

What I'm really asking

I think my biggest problem right now isn't a single error or a specific algorithm.

It's that I'm struggling to understand how to make sound technical decisions throughout an ML project.

I keep discovering new approaches, questioning previous choices, and wondering whether I'm solving the problem correctly in the first place.

If you were reviewing a project like this, what would you look at first?

How would you decide what is necessary, what is overengineering, and what should be removed?

I'm not expecting anyone to design the entire system for me. I'd genuinely like to understand the reasoning behind these decisions.

If you have any suggestions, resources, or lessons from your own projects, I'd really appreciate them.

And if you've ever started a project with an ambitious technical idea, only to end up questioning the entire approach halfway through, I'd love to hear how you navigated it.

Thanks for reading!