[nudpiedo]

Thanks for the offering, seriously, it is an important investment/cost I wasn't sure it would be worth right now. (If you wish to I added some contact information, for an easier chat, in my HN profile; just for the following day).

Here is a realistic case I would have used

Short prompt: "research the market of the SysML products and industry cap, key actors, opportunities and advantages"

Expanded prompt:

“Conduct a detailed market analysis of SysML (Systems Modeling Language) products and services, focusing on the following aspects: 1. Industry Overview: • Current market size and growth trends for SysML-related products and services. • Global market cap or valuation of the SysML industry or similar system engineering software. 2. Key Players: • Identify major companies and organizations offering SysML tools, including established leaders and emerging competitors. • Provide a brief description of each key player’s products, innovations, and market share. 3. Product Offerings: • Highlight the range of SysML-based products available (e.g., modeling software, integration tools, training services). • Compare features, target industries, and pricing strategies. 4. Market Opportunities: • Explore new or underserved industries where SysML adoption could grow (e.g., aerospace, automotive, healthcare). • Identify gaps in current product offerings that represent potential areas for innovation or competitive advantage. 5. Competitive Advantages: • Analyze what makes SysML valuable to companies (e.g., improving system complexity management, ensuring design consistency). • Evaluate how SysML tools offer a competitive advantage compared to alternative solutions like UML or other system modeling methods. 6. Challenges and Risks: • Discuss potential challenges such as market saturation, training requirements, or competing technologies. • Highlight external factors that could affect market growth, such as regulations or advances in adjacent industries.

Provide sources where relevant, including reports, studies, or insights from industry experts.”

Hvala.

[A_D_E_P_T]

Report link: https://chatgpt.com/share/67a4a67b-5060-8005-85b8-65eef3cb60...

Let me know what you think, will ya? I'm curious as to how you'd evaluate the quality of that report.

Also note the follow-up prompt I gave it. This thing needs as much detail as you can give it, and small changes to your prompt can hugely influence the end result.

[nudpiedo]

Wow, thank you so much, that was 11 pages full pages. I will be a bit critical with it hoping my review helps you as well, since to me, it was very friendly and useful your offer (also I wonder if you can still do 99 more reserachs like this one during the month or this amounts already to many of such research in total).

Superficially:

- Awesome how fast is available and all sources linked to explain each claim

- To export that to a proper doc with footnotes and proper formatting it's already something to be worked by openAI

- it looks like the perfect way to create a gut feeling and a sense of what is going on

Content:

- Wrong studies: It mixed SysML (a particular visual language for which it was specifically requested) with MBSE (the family of tools) which is exactly not the same as the desired study was particular for SysML.

- Quality of data: Most of the data comes from public articles and studies made by others, all the time about MBSE, not SysML, and just quotes their numbers, it does not do its own estimates looking for the benefits of such products on each company and estimating a projection (that would be an actual research, and an AI should be capable of tirelessly do that or even biasedly look for the right pieces if information). For example it was a report on diet, a report like that should avoid debunk articles, bro's blogs, etc.

- Inconsistent scales: at some comparison table, it mentioned at the foot that it will display pricing with such schema:(Pricing: $ = low, $$$$ = high) however it made that in a single row. Why? the source for that field made that as well in its source, but none of the other fields repeated this system to value or adapt results.

- Only googleable data: companies reports or private databases here are key for a high quality report. Sometimes this is not always possible for an AI or a crawler but here am I evaluating the outcome (use case): a market analisis for strategic purposes.

- Quality of the report: Many things mentioned like services around the products are also highly valuable... would be useful to remark case by case the business model of each company and how much is in the product and how much is in the related service (using a pie chats or whatever) and showing particular case studies to remark the market trend, from which model is coming from (product) and where it is going to (services and SaaS).

I could continue longer with many other things and such errors but it is quite long.

Conclusion:

It is very useful, particularly to grasp a general, yet detailed, idea on what is going on, on a market. However it is only as valid as a remix of previous things, not an actual market research for an actual strategy. Many sources, elements, landscape of which companies and products related are there are totally useful, perhaps 30%-40% of the total work and it gives a clear structure where to go from here.

Probably it may improve the more interactive that the tool is, for example asking to correct some sections or improve in specifically suggested ways by the user. Basically the user needs to bring expertise, reasoning from the field and critical thinking (things machines will be lacking in any foreseeable future).

Why am I so critic? Remember, map is not the territory, particularly in strategic terms. And it is also a problem that many professionals do also these kind of failures: uncritically copying data from other's reports without verifying critically any of it which leads to very specific kinds of strategic errors.

It will become more useful as it becomes specialized in the kind of reports, and judges critically (which does not) and if it can be adapted to work on a private repo or preselected amounts of sources, or even prescripted agent behaves for the sort of report.

Verdict:

I would purchase it, not to solve the problem or resell it, but as a way to get started and accelerate the process. It already does what an internship student would do, or a mediocre professional: revamp preexisting mashups to get a general, but detailed, feeling but no more insights or research than what it is already well known (googling after all).

It has a great future as it would be great if such level of non creative work is automated away as their value often is marginal and uncritically propagates previous beliefs and biases (and if there is a centralized tool, that can be tuned to avoid well known issues)

[A_D_E_P_T]

> It is very useful, particularly to grasp a general, yet detailed, idea on what is going on, on a market. However it is only as valid as a remix of previous things, not an actual market research for an actual strategy. Many sources, elements, landscape of which companies and products related are there are totally useful, perhaps 30%-40% of the total work and it gives a clear structure where to go from here.

> Probably it may improve the more interactive that the tool is, for example asking to correct some sections or improve in specifically suggested ways by the user. Basically the user needs to bring expertise, reasoning from the field and critical thinking (things machines will be lacking in any foreseeable future).

Yeah, that's just the thing. With what you know, you can iterate on the results it gives you. It's very sensitive to how your prompt is written and structured, so some fine tuning, user-provided context, and user expertise, it'll dial-in on any subject very well. It's not top-expert-level yet -- at least not on its own -- but it's close, and it's miles better than asking o1-Pro (or Deepseek r1) for a detailed report.