Uploading large files in java

Upload large files : Spring Boot

So guys, I was dealing with a problem recently. I was getting OutOfMemoryError when trying to upload and save large files (like 2/3 gbs). I was trying to deal it with HttpServletRequest but didn’t end so well. But after spending some time thinking about the universe, mens style, water pond (road? seriously. ) on dhaka city after a heavy rain and how to make life easier doing absolutely nothing, figured out the nicest way to do it.

We’ll use apache commons IO to copy inputstream (and write) to a file. But it has nothing to do with OutOfMemoryError. It’s just a convenient and simple way to write inputstream to a file

1. Dependency

2. Create a multipart form

 
 
Image
 
 

3. Controller

@RequestMapping(value = "/create", method = RequestMethod.POST) private String create(@RequestParam("image") MultipartFile multipartFile) throws IOException< /*** * Don't do this. * it loads all of the bytes in java heap memory that leads to OutOfMemoryError. We'll use stream instead. **/ // byte[] fileBytes = multipartFile.getBytes(); InputStream fileStream = multipartFile.getInputStream(); File targetFile = new File("src/main/resources/targetFile.tmp"); FileUtils.copyInputStreamToFile(fileStream, targetFile); return "redirect:/?message=Successful!"; >

Nice. Everything should be alright and your file should stay in right place without braking of corrupting your data.

Share this:

Источник

Upload Large File in a Spring Boot 2 Application Using Swagger UI

Join the DZone community and get the full member experience.

In this article, we are going to create a sample Spring Boot application for uploading large files using Swagger UI. The API created for uploading large files can receive an HTTP multi-part file upload request.

What We Will Need to Build the Application

• JDK 1.8 or later
• Maven 3.2+
• A favorite text editor or IDE. You can also import this example code straight into your IntelliJ IDEA from GitHub. The GitHub link is provided at the end of the article.
• Spring Boot 2.0.1. RELEASE
• Spring Framework 5.0.5. RELEASE
• Spring Cloud Consul 2.0.0.M5
• Swagger 2.8.0

Consul is used to hold the externalized properties for our Spring Boot application. Read about Consul integration with Spring Boot 2

Step 1: Download Consul Agent. Once the download is complet, go to the specific directory and use the following command to start the agent.

consul agent -dev -config-dir=C:\\consul_0.7.3_windows_amd64

Once the agent is started, the Consul Agent UI can be accessed via this URL: http://localhost:8500/ui/#

Step 2: Now, create your application’s YAML file with configurations under the key-value section of Consul UI (config/spring-boot-file-upload.yml).

The content of this file is :

spring: application: name: spring-boot-file-upload servlet: multipart: enabled: false max-file-size: 10MB server: servlet: contextPath: /spring-boot-file-upload port: 8090 logging: level: ROOT: DEBUG

Now let us revisit the following configurations from the above file:

servlet.multipart.enabled: false servlet.multipart.max-file-size: 10MB

For large files, we cannot use Spring Boot’s default StandardServletMultipartResolver or CommonsMultipartResolver , since the server has limited resources (disk space) or memory for buffering. So we need to disable the default MultipartResolver and define our own MultipartResolver , which is present in the main application class.

We need to provide the Consul information to our Spring Boot application via the bootstrap.yml file, which is present in the resources folder of the project.

spring: application: name: spring-boot-file-upload cloud: consul: host: localhost port: 8500 discovery: tags: spring-boot-file-upload enabled: true config: enabled: true format: files fail-fast: true

Project Structure:

The standard project structure is as follows:

Create an Application Class

To start a Spring Boot MVC application, we first need a starter. Thanks to Spring Boot, everything is auto-configured for you. All you need to get started with this application is the following SpringBoot2FileUpload class.

package com.tuturself; import lombok.extern.slf4j.Slf4j; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.EnableAutoConfiguration; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.cloud.client.discovery.EnableDiscoveryClient; import org.springframework.context.annotation.Bean; import org.springframework.web.multipart.commons.CommonsMultipartResolver; /** * Created by Arpan Das on 4/17/2018. */ @Slf4j @EnableDiscoveryClient @SpringBootApplication @EnableAutoConfiguration public class SpringBoot2FileUpload < public static void main(String[] args) < log.info("SpringBoot2FileUpload Application is Starting. "); try < SpringApplication.run(SpringBoot2FileUpload.class, args); >catch (Exception e) < log.error("Error occurred while starting SpringBoot2FileUpload"); >log.info("SpringBoot2FileUpload Application Started.."); > @Bean(name = "multipartResolver") public CommonsMultipartResolver multipartResolver() < CommonsMultipartResolver multipartResolver = new CommonsMultipartResolver(); multipartResolver.setMaxUploadSize(-1); return multipartResolver; >>

As described earlier, we have defined our multipartResolver in our main application class SpringBoot2FileUpload.java and set the MaxUploadSize property to -1. Now it will read the maximum file size from the spring-boot-file-upload.yml file in Consul, which is set to 10MB.

Create a File Upload Controller

Now let us create the Controller class, FileUploadRestApi.java , where we have defined the REST API to upload the file. The API is documented using Swagger UI.

package com.tuturself.webservice; import io.swagger.annotations.*; import org.apache.commons.io.FileUtils; import org.springframework.http.HttpStatus; import org.springframework.http.MediaType; import org.springframework.http.ResponseEntity; import org.springframework.web.bind.annotation.PostMapping; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RequestPart; import org.springframework.web.bind.annotation.RestController; import org.springframework.web.multipart.MultipartFile; import java.io.File; import java.io.IOException; import java.util.List; @RestController @RequestMapping("/upload") @Api(value = "Guidelines", description = "Describes the guidelines for " + " Spring boot 2.0.1 for uploading large file using Swagger UI") public class FileUploadRestApi < @PostMapping @ApiOperation(value = "Make a POST request to upload the file", produces = "application/json", consumes = MediaType.MULTIPART_FORM_DATA_VALUE) @ApiResponses(value = < @ApiResponse(code = 200, message = "The POST call is Successful"), @ApiResponse(code = 500, message = "The POST call is Failed"), @ApiResponse(code = 404, message = "The API could not be found") >) public ResponseEntity uploadFile( @ApiParam(name = "file", value = "Select the file to Upload", required = true) @RequestPart("file") MultipartFile file) < try < File testFile = new File("test"); FileUtils.writeByteArrayToFile(testFile, file.getBytes()); Listlines = FileUtils.readLines(testFile); lines.forEach(line -> System.out.println(line)); > catch (IOException e) < e.printStackTrace(); return new ResponseEntity("Failed", HttpStatus.INTERNAL_SERVER_ERROR); > return new ResponseEntity("Done", HttpStatus.OK); > >

Creating the Swagger Configuration Class

We need to configure Swagger UI with our Spring Boot 2 application.

package com.tuturself.config; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import springfox.documentation.builders.ApiInfoBuilder; import springfox.documentation.builders.PathSelectors; import springfox.documentation.builders.RequestHandlerSelectors; import springfox.documentation.service.ApiInfo; import springfox.documentation.service.Contact; import springfox.documentation.spi.DocumentationType; import springfox.documentation.spring.web.plugins.Docket; import springfox.documentation.swagger2.annotations.EnableSwagger2; /** * @author Arpan Das */ @Configuration @EnableSwagger2 public class SwaggerConfig < @Bean public Docket api() < return new Docket(DocumentationType.SWAGGER_2) .select() .apis(RequestHandlerSelectors.basePackage("com.tuturself.webservice")) .paths(PathSelectors.any()) .build().apiInfo(metaData()) .useDefaultResponseMessages(false); >private ApiInfo metaData() < return new ApiInfoBuilder() .title("Spring Boot 2.0 File Upload example with Consul Integration & Swagger 2.8.0") .description("Upload file Swagger-ui 2.8.0 and Spring Boot 2 Spring Cloud Consul") .version("version 1.0") .contact(new Contact("Tutu'rself", "https://www.tuturself.com", "arpan.kgp@gmail.com")).build(); >>

Now the application is ready. Just check for all the required dependencies in the pom.xml file.

  4.0.0 com.tuturself spring-boot-file-upload 1.0-SNAPSHOT org.springframework.boot spring-boot-starter-parent 2.0.1.RELEASE  1.16.20 5.0.5.RELEASE 2.0.0.M5 2.8.0 UTF-8 UTF-8 1.8    org.springframework.boot spring-boot-starter-web  org.springframework spring-context $ org.springframework.boot spring-boot-starter-test test  org.springframework.cloud spring-cloud-starter-consul-all $ org.springframework.cloud spring-cloud-consul-discovery $   io.springfox springfox-swagger2 $ io.springfox springfox-swagger-ui $   org.projectlombok lombok $ provided  commons-fileupload commons-fileupload 1.3.3   org.apache.commons commons-io 1.3.2    org.apache.maven.plugins maven-compiler-plugin 3.5.1 $ $ $  org.springframework.boot spring-boot-maven-plugin true     src/main/resources true     spring-milestones Spring Milestones http://repo.spring.io/milestone false      org.springframework.cloud spring-cloud-dependencies Finchley.M7 pom import    

Now run the application. The Swagger UI app can be accessed via the URL, http://localhost:8080/spring-boot-consul/swagger-ui.html#

To try the POST API for uploading a file, click on it. Once it is expanded, click on Try Now and then choose a file to upload and Execute.

The project can be downloaded from GitHub: spring-boot-file-upload.

Published at DZone with permission of Arpan Das . See the original article here.

Opinions expressed by DZone contributors are their own.

Источник

Saved searches

Use saved searches to filter your results more quickly

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

Upload big file by java. Using Apache HttpComponents. Can work on Android too.

License

clxy/BigFileUploadJava

This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Sign In Required

Please sign in to use Codespaces.

Launching GitHub Desktop

If nothing happens, download GitHub Desktop and try again.

Launching GitHub Desktop

If nothing happens, download GitHub Desktop and try again.

Launching Xcode

If nothing happens, download Xcode and try again.

Launching Visual Studio Code

Your codespace will open once ready.

There was a problem preparing your codespace, please try again.

Latest commit

Git stats

Files

Failed to load latest commit information.

README.md

Upload big file using java.

Basically, read a big file into small parts and upload. When all file parts upload is complete, combine at server.

1. Read the big file into several small parts. Considering I/O contention, use just one thread; Considering memory usage, read file part by part into fixed size queue.
2. Upload every readed file part. Usually multiple threads would be better, but can’t too much, default threads count is 5.
3. After all parts uploaded, notify server to combine.
4. Can retry the specific parts only if failed to process.

• Producer/Consumer pattern.
• Communicate read and upload processes by BlockingQueue.
• Uploading is using Apache HttpComponents currently. There can be other implementations.
• Can be used in the android. Please refer to BigFileUploadAndroid。

Please note that the maximum memory usage might be: PART_SIZE * (MAX_UPLOAD + MAX_READ — 1)

UploadFileService service = new UploadFileService(yourFileName); service.upload(); 

UploadFileService service = new UploadFileService(yourFileName); service.retry(1, 2); 

Because it is via HTTP, so it can be in any language, such as Java, PHP, Python, etc. Here is a java example.

. try (FileOutputStream dest = new FileOutputStream(destFile, true)) < FileChannel dc = dest.getChannel();// the final big file. for (long i = 0; i < count; i++) < File partFile = new File(destFileName + "." + i);// every small parts. if (!partFile.exists()) < break; > try (FileInputStream part = new FileInputStream(partFile)) < FileChannel pc = part.getChannel(); pc.transferTo(0, pc.size(), dc);// combine. > partFile.delete(); > statusCode = OK;// set ok at last. > catch (Exception e) < log.error("combine failed.", e); >

Источник