[backend/frontend] feat(catalog): add catalog connector ingestion data (#4311)

[camrrx]

Proposed changes

• Ingestion json connectors file
• Display catalog page

Testing Instructions

1. Go on catalog page
2. You should have access to all the connectors

Related issues

• Closes [EE] Implement the catalog (UI+Backend) #4311

Checklist

• I consider the submitted work as finished
• I tested the code for its functionality
• I wrote test cases for the relevant uses case
• I added/update the relevant documentation (either on github or on notion)
• Where necessary I refactored code to improve the overall quality
• For bug fix -> I implemented a test that covers the bug

Further comments

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc...

[camrrx]

We don't have default image yet

[antoinemzs]

Suggestion: to keep source code cleaner (or more "magical"), Lombok provides class annotations to hide constructors such as this one which only assigns instance fields.

@RequiredArgsConstructor on the class allows you to delete this constructor (Lombok will autogenerate it).

Suggested change

	@Slf4j
	@Service
	@Transactional
	public class CatalogConnectorIngestionService {
	private static final Set<String> PROTECTED_KEYS = Set.of(
	"COLLECTOR_ID",
	"INJECTOR_ID",
	"EXECUTOR_ID"
	);
	private final CatalogConnectorService catalogConnectorService;
	private final FileService fileService;
	private final ConnectorInstanceService connectorInstanceService;
	private final ConnectorInstanceConfigurationRepository connectorInstanceConfigurationRepository;
	public CatalogConnectorIngestionService(
	CatalogConnectorService catalogConnectorService,
	FileService fileService,
	ConnectorInstanceService connectorInstanceService,
	ConnectorInstanceConfigurationRepository connectorInstanceConfigurationRepository) {
	this.catalogConnectorService = catalogConnectorService;
	this.fileService = fileService;
	this.connectorInstanceService = connectorInstanceService;
	this.connectorInstanceConfigurationRepository = connectorInstanceConfigurationRepository;
	}
	@Slf4j
	@Service
	@Transactional
	@RequiredArgsConstructor
	public class CatalogConnectorIngestionService {
	private static final Set<String> PROTECTED_KEYS = Set.of(
	"COLLECTOR_ID",
	"INJECTOR_ID",
	"EXECUTOR_ID"
	);
	private final CatalogConnectorService catalogConnectorService;
	private final FileService fileService;
	private final ConnectorInstanceService connectorInstanceService;
	private final ConnectorInstanceConfigurationRepository connectorInstanceConfigurationRepository;

[antoinemzs]

Note: no error handling here. A lot of things can go wrong in deserialisation situations, so are we prepared for an all-or-nothing ingestion (because the process is transactional) ?
I'm asking because functionally, what are we expecting to see if there is, for example, a single contract throwing an error?

[camrrx]

To confirm with @jborozco !

[antoinemzs]

it seems like this would have the potential for collisions, but not terribly dangerous apart if we accidentally upload the same path elsewhere in the codebase.

Suggestion: using the DB-generated ID for the persisted catalog connector object as one of the filename fragments.

This would have you remove the use of .saveAll(List<CatalogConnector>) in favour of a loop with .save(CatalogConnector) in extractCatalog; but that's OK because we are already within a transaction scope (the overhead of using a loop+save is to create a new transaction scope for each iteration)

@Override
@Transactional
public <S extends T> List<S> saveAll(Iterable<S> entities) {

	Assert.notNull(entities, ENTITIES_MUST_NOT_BE_NULL);

	List<S> result = new ArrayList<>();

	for (S entity : entities) {
		result.add(save(entity));
	}

	return result;
}

[camrrx]

Good suggestion, as discuss we'll not do it now.

[antoinemzs]

good thinking

[antoinemzs]

It looks like we should make an effort to encrypt the currently plaintext value with the known XTM Composer public key instead of nuking the config line altogether. What are your thoughts?

[camrrx]

Todo : another PR when the method to encrypt will be merged

[antoinemzs]

Is this correct? This looks like this could cause issues down the line. What are the impacts of this occurrence?

[camrrx]

We keep it like this as discussed

[antoinemzs]

should we be concerned that perhaps there is an entry in [required] that doesn't match any listed properties? Thoughts?

Note this would meant the contract is malformed. Consider adding some logging.

[codecov]

Codecov Report

❌ Patch coverage is 51.32743% with 110 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.89%. Comparing base (267049f) to head (4f4f5c4).
⚠️ Report is 1 commits behind head on release/current.

Files with missing lines	Patch %	Lines
...g_connectors/CatalogConnectorIngestionService.java	63.09%	42 Missing and 20 partials ⚠️
...o/openaev/utils/mapper/CatalogConnectorMapper.java	6.66%	14 Missing ⚠️
...ev/rest/catalog_connector/CatalogConnectorApi.java	0.00%	11 Missing ⚠️
...penaev/runner/CatalogConnectorIngestionRunner.java	0.00%	11 Missing ⚠️
...-api/src/main/java/io/openaev/utils/TimeUtils.java	0.00%	6 Missing ⚠️
...ce/catalog_connectors/CatalogConnectorService.java	33.33%	4 Missing ⚠️
...tor_instance/service/ConnectorInstanceService.java	33.33%	2 Missing ⚠️

Additional details and impacted files

@@                 Coverage Diff                 @@
##             release/current    #4379    +/-   ##
===================================================
  Coverage              50.89%   50.89%            
- Complexity              3762     3785    +23     
===================================================
  Files                    914      919     +5     
  Lines                  27396    27621   +225     
  Branches                2059     2098    +39     
===================================================
+ Hits                   13942    14058   +116     
- Misses                 12608    12697    +89     
- Partials                 846      866    +20     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.