Listing AWS S3 bucket content from Springboot With Apache Camel

Question

I have a sprintboot project running v2.5.4 which works fine. I have access to S3 and im able to list the content of a bucket i have created. So i wanted to experiment with Apache Camel to try to just list the content of a bucket which should be pretty simple according to the examples. But i keep getting errors.

I added 2 dependencies to my build.gradle

implementation group: 'org.apache.camel.springboot', name: 'camel-core-starter', version: '3.13.0'
implementation group: 'org.apache.camel.springboot', name: 'camel-aws2-s3-starter', version: '3.13.0'

and then i simply created a SimpleRouteBuilder.java

@Component

public class SimpleRouteBuilder extends RouteBuilder {

@Override
public void configure() throws Exception {
    from("aws2-s3://bucketName?amazonS3Client=#createS3Client&operation=listObjects&accessKey=xxxAccessKeyxxx&secretKey=xxxSecretKeyxxx")
            .log("Received body: ");
}

And i keep getting this stacktrace

On my aws s3 client factory i have set the bean name

@Slf4j
@Configuration
public class S3ClientBeanFactory {

    @Bean(name = "s3Client")

and this seems to work - when i change the name to something else i get an error about this :

No bean could be found in the registry for:S3Client

But with the "s3client" set in the camel endpoint url i get this all the time

2021-12-13 12:23:25.036  INFO [,,] 28267 --- [           main] o.a.c.impl.engine.AbstractCamelContext   : Apache Camel 3.13.0 (camel-1) shutdown in 4ms (uptime:511ms)
2021-12-13 12:23:25.045  INFO [,,] 28267 --- [           main] o.apache.catalina.core.StandardService   : Stopping service [Tomcat]
2021-12-13 12:23:25.069  INFO [,,] 28267 --- [           main] ConditionEvaluationReportLoggingListener : 
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2021-12-13 12:23:25.085 ERROR [,,] 28267 --- [           main] o.s.boot.SpringApplication               : Application run failed
org.apache.camel.FailedToStartRouteException: Failed to start route route1 because of null
2021-12-13 12:23:25.036  INFO [,,] 28267 --- [           main] o.a.c.impl.engine.AbstractCamelContext   : Apache Camel 3.13.0 (camel-1) shutdown in 4ms (uptime:511ms)
2021-12-13 12:23:25.045  INFO [,,] 28267 --- [           main] o.apache.catalina.core.StandardService   : Stopping service [Tomcat]
2021-12-13 12:23:25.069  INFO [,,] 28267 --- [           main] ConditionEvaluationReportLoggingListener : 
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2021-12-13 12:23:25.085 ERROR [,,] 28267 --- [           main] o.s.boot.SpringApplication               : Application run failed
org.apache.camel.FailedToStartRouteException: Failed to start route route1 because of null
    at org.apache.camel.impl.engine.RouteService.warmUp(RouteService.java:123)
    at org.apache.camel.impl.engine.InternalRouteStartupManager.doWarmUpRoutes(InternalRouteStartupManager.java:306)
    at org.apache.camel.impl.engine.InternalRouteStartupManager.safelyStartRouteServices(InternalRouteStartupManager.java:189)
    at org.apache.camel.impl.engine.InternalRouteStartupManager.doStartOrResumeRoutes(InternalRouteStartupManager.java:147)
    at org.apache.camel.impl.engine.AbstractCamelContext.doStartCamel(AbstractCamelContext.java:3201)
    at org.apache.camel.impl.engine.AbstractCamelContext.doStartContext(AbstractCamelContext.java:2863)
    at org.apache.camel.impl.engine.AbstractCamelContext.doStart(AbstractCamelContext.java:2814)
    at org.apache.camel.spring.boot.SpringBootCamelContext.doStart(SpringBootCamelContext.java:43)
    at org.apache.camel.support.service.BaseService.start(BaseService.java:119)
    at org.apache.camel.impl.engine.AbstractCamelContext.start(AbstractCamelContext.java:2510)
    at org.apache.camel.impl.DefaultCamelContext.start(DefaultCamelContext.java:246)
    at org.apache.camel.spring.SpringCamelContext.start(SpringCamelContext.java:119)
    at org.apache.camel.spring.SpringCamelContext.onApplicationEvent(SpringCamelContext.java:151)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:421)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:378)
    at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:938)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:338)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1332)
    at dk.danskespil.scratchgames.ScratchgamesApplication.main(ScratchgamesApplication.java:22)
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 403, Request ID: null, Extended Request ID: FknaUW6/yRkYvJry9d8oIWU2hC4aRk7z8ilAZZxlcDN4s+P4bAoyzWVriJxUYj2bCyzCFFMSGNY=)

Is this operation not possible or what am i missing to do such simple operation ?

Answer 1

Spring doesn't consider configuration beans to be special. It is possible that the route bean is created before the S3 client bean.

I'd try using the @DependsOn({"s3ClientBeanFactory"})-annotation on the route bean. More on it here: Controlling Bean Creation Order with @DependsOn Annotation

Answer 2

I ran into the same issue with Apache Camel component aws2-s3:// which was caused by insufficient rights on AWS S3:

Caused by: software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 403, ...)

But I have to mention that S3Client from Amazon SDK works well for reading files with the same rights = the same account.

Explanation: I found that this aws2-s3:// component requires making headBucket API call (and the others) which causes the error because of insufficient rights for making this api call.