Compare commits

...

10 Commits

Author SHA1 Message Date
GiteaBot
6b8dd90dc7 [skip ci] Updated translations via Crowdin 2026-05-02 01:02:44 +00:00
silverwind
abcfa53040 Replace olivere/elastic with REST API client, add OpenSearch support (#37411)
Drops `github.com/olivere/elastic/v7` (unmaintained) and replaces it
with a small in-house wrapper that speaks the Elasticsearch REST API
directly via `net/http`. The subset used by Gitea (`_cluster/health`,
`_bulk`, `_doc`, `_delete_by_query`, `_refresh`, `_search`, `HEAD`/`PUT`
index) is stable across the targeted servers, so no client library is
needed.

**Targets tested**
- Elasticsearch 7, 8, 9
- OpenSearch 1, 2, 3

**Why not `go-elasticsearch`?**
The official client enforces an `X-Elastic-Product` server-identity
check that OpenSearch deliberately fails, which would force shipping a
transport shim to defeat it. Going direct over `net/http` removes that
fight along with several MB of transitive deps (`elastic-transport-go`,
`go.opentelemetry.io/otel{,/metric,/trace}`, `auto/sdk`, `easyjson`,
`intern`, `logr`, `stdr`).

Replaces: #30755
Fixes: https://github.com/go-gitea/gitea/issues/30752

---
This PR was written with the help of Claude Opus 4.7

---------

Co-authored-by: Claude (Opus 4.7) <noreply@anthropic.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2026-05-02 00:12:54 +02:00
silverwind
31cee60cc7 Improve code editor text selection and clean up lint enablement (#37474)
1. Make the content area stretch the box, enabling text selection to
start over empty space.
2. Disable linter for markdown, it can never produce lint errors, this
hides the unnecessary lint gutter on markdown files.
3. Verified all languages linter enablement, all accurate.
4. Refactor `getLinterExtension` to not rely on file extensions.
5. Include jsonc/json5 extensions in regex.

---
This PR was written with the help of Claude Opus 4.7

---------

Co-authored-by: Claude (Opus 4.7) <noreply@anthropic.com>
Co-authored-by: Nicolas <bircni@icloud.com>
2026-05-01 17:41:31 +00:00
wxiaoguang
deb31d3f30 Refactor database connection (#37496)
Clean up legacy copied&pasted code, introduce the unique "database
connection" function. Move migration testing helper function
PrepareTestEnv to a separate package.

By the way, remove "shadow connection secrets" tricks: showing
connection string on UI is useless

---------

Co-authored-by: Nicolas <bircni@icloud.com>
2026-05-01 15:38:38 +00:00
pomidorry
02b1b8a549 Add mirror auth updates to repo edit API and settings (#37468)
## Summary

This PR adds support for updating pull mirror authentication via the
repository edit API and UI.

It introduces new mirror authentication fields in _EditRepoOption_,
updates the API logic to safely handle partial credential updates, and
fixes the web settings flow so that the existing remote username is
preserved when only the password is changed.

### What changed
- added _auth_username_, _auth_password_, and _auth_token_ to
EditRepoOption
- updated the repository edit API to apply mirror auth changes via
_updateMirror_
- preserved existing username/password when only part of the auth
payload is provided
- used oauth2 as the default username when _auth_token_ is provided
- kept stored mirror URLs sanitized in DB and API responses
- updated Swagger schema for the new API fields
- added API integration tests for password-only and token-only updates
- added a web settings test to ensure username preservation on partial
updates

## Why

Some use cases require automated synchronization of pull mirrors, for
example in CI/CD pipelines or integrations with external systems.

At the same time, many organizations enforce security policies that
require periodic token rotation (e.g., monthly).

Currently, mirror credentials can only be updated via the UI, which
makes automation difficult.

## This change enables:

- automated token rotation
- avoiding manual updates via the UI
- easier integration with secret management systems
## Testing
- added integration coverage for mirror auth updates via _PATCH
/api/v1/repos/{owner}/{repo}_
- added web settings tests for password-only updates preserving the
existing username

## Result
Ability to automate auth update
<img width="2400" height="1245" alt="1"
src="https://github.com/user-attachments/assets/67fd5cca-9cb3-4536-b0e2-4d09b8ebff0f"
/>
<img width="962" height="932" alt="image"
src="https://github.com/user-attachments/assets/5d548f5d-aadf-4807-ba52-9c29df93a4cc"
/>

Generative AI was used to help with making this PR.
##
2026-05-01 11:00:03 +00:00
Lunny Xiao
48cea1fb79 Fix basic auth bug (#37486) 2026-04-30 20:34:43 -07:00
wxiaoguang
1721c235a7 Refactor CI workflows (#37487)
1. only trigger docker-dryrun arm64&riscv64 when dockerfile changes
2. de-duplicate "contents: read" permission for most workflows
3. merge various "lint-*" jobs into one job
4. add missing lint targets to the "lint" (all) target
2026-05-01 02:15:01 +08:00
Icy Avocado
81692ceafa Allow multiple projects per issue and pull requests (#36784)
Add ability to add and remove multiple projects per issue
and pull request.

Resolve #12974

---------

Signed-off-by: Icy Avocado <avocado@ovacoda.com>
Co-authored-by: Tyrone Yeh <siryeh@gmail.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: OpenCode (gpt-5.2-codex) <opencode@openai.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Claude (Opus 4.7) <noreply@anthropic.com>
2026-04-30 22:38:05 +08:00
GiteaBot
52d6baf5a8 [skip ci] Updated translations via Crowdin 2026-04-30 01:05:39 +00:00
wxiaoguang
2b2ec6af85 Refactor compare diff/pull page (1) (#37481)
1. Rename CompareInfo.MergeBase to CompareBase, it is not merge base
2. Remove unused template variables `ctx.Data["Username"]` and
`ctx.Data["Reponame"]`
3. Decouple some template variable accesses, use typed struct

---------

Co-authored-by: Nicolas <bircni@icloud.com>
2026-04-29 18:32:46 +00:00
164 changed files with 3603 additions and 2057 deletions

View File

@@ -0,0 +1,29 @@
name: docker-dryrun
description: Composite action that performs the container build steps for a single platform.
inputs:
platform:
description: "The target platform: linux/amd64, linux/arm64, linux/riscv64."
required: true
runs:
using: composite
steps:
- uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v4.0.0
- uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
- name: Build regular image
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
with:
context: .
platforms: ${{ inputs.platform }}
push: false
file: Dockerfile
cache-from: type=registry,ref=ghcr.io/go-gitea/gitea:buildcache-rootful
- name: Build rootless image
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
with:
context: .
platforms: ${{ inputs.platform }}
push: false
file: Dockerfile.rootless
cache-from: type=registry,ref=ghcr.io/go-gitea/gitea:buildcache-rootless

View File

@@ -27,11 +27,12 @@ concurrency:
group: cache-seeder group: cache-seeder
cancel-in-progress: true cancel-in-progress: true
permissions:
contents: read
jobs: jobs:
gobuild: gobuild:
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions:
contents: read
steps: steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0 - uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
@@ -49,8 +50,6 @@ jobs:
lint: lint:
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions:
contents: read
strategy: strategy:
fail-fast: false fail-fast: false
matrix: matrix:

View File

@@ -11,13 +11,14 @@ concurrency:
env: env:
RENOVATE_VERSION: 43.141.5 # renovate: datasource=docker depName=ghcr.io/renovatebot/renovate RENOVATE_VERSION: 43.141.5 # renovate: datasource=docker depName=ghcr.io/renovatebot/renovate
permissions:
contents: read
jobs: jobs:
cron-renovate: cron-renovate:
runs-on: ubuntu-latest runs-on: ubuntu-latest
if: github.repository == 'go-gitea/gitea' # prevent running on forks if: github.repository == 'go-gitea/gitea' # prevent running on forks
timeout-minutes: 30 timeout-minutes: 30
permissions:
contents: read
steps: steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: renovatebot/github-action@83ec54fee49ab67d9cd201084c1ff325b4b462e4 # v46.1.10 - uses: renovatebot/github-action@83ec54fee49ab67d9cd201084c1ff325b4b462e4 # v46.1.10

View File

@@ -15,6 +15,8 @@ on:
value: ${{ jobs.detect.outputs.templates }} value: ${{ jobs.detect.outputs.templates }}
docker: docker:
value: ${{ jobs.detect.outputs.docker }} value: ${{ jobs.detect.outputs.docker }}
dockerfile:
value: ${{ jobs.detect.outputs.dockerfile }}
swagger: swagger:
value: ${{ jobs.detect.outputs.swagger }} value: ${{ jobs.detect.outputs.swagger }}
yaml: yaml:
@@ -24,12 +26,13 @@ on:
e2e: e2e:
value: ${{ jobs.detect.outputs.e2e }} value: ${{ jobs.detect.outputs.e2e }}
permissions:
contents: read
jobs: jobs:
detect: detect:
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 3 timeout-minutes: 3
permissions:
contents: read
outputs: outputs:
backend: ${{ steps.changes.outputs.backend }} backend: ${{ steps.changes.outputs.backend }}
frontend: ${{ steps.changes.outputs.frontend }} frontend: ${{ steps.changes.outputs.frontend }}
@@ -37,6 +40,7 @@ jobs:
actions: ${{ steps.changes.outputs.actions }} actions: ${{ steps.changes.outputs.actions }}
templates: ${{ steps.changes.outputs.templates }} templates: ${{ steps.changes.outputs.templates }}
docker: ${{ steps.changes.outputs.docker }} docker: ${{ steps.changes.outputs.docker }}
dockerfile: ${{ steps.changes.outputs.dockerfile }}
swagger: ${{ steps.changes.outputs.swagger }} swagger: ${{ steps.changes.outputs.swagger }}
yaml: ${{ steps.changes.outputs.yaml }} yaml: ${{ steps.changes.outputs.yaml }}
json: ${{ steps.changes.outputs.json }} json: ${{ steps.changes.outputs.json }}
@@ -94,6 +98,10 @@ jobs:
- "docker/**" - "docker/**"
- "Makefile" - "Makefile"
dockerfile:
- "Dockerfile"
- "Dockerfile.rootless"
swagger: swagger:
- "templates/swagger/v1_json.tmpl" - "templates/swagger/v1_json.tmpl"
- "templates/swagger/v1_input.json" - "templates/swagger/v1_input.json"

View File

@@ -1,35 +0,0 @@
# Reusable workflow that performs the container build steps for a single platform.
# Used by `pull-docker-dryrun.yml` to run builds in parallel per-platform.
on:
workflow_call:
inputs:
platform:
description: 'The target platform(s) to build for (e.g. linux/amd64)'
required: true
type: string
jobs:
build-dryrun:
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v4.0.0
- uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
- name: Build rootful image
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
with:
context: .
platforms: ${{ inputs.platform }}
push: false
file: Dockerfile
cache-from: type=registry,ref=ghcr.io/go-gitea/gitea:buildcache-rootful
- name: Build rootless image
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
with:
context: .
platforms: ${{ inputs.platform }}
push: false
file: Dockerfile.rootless
cache-from: type=registry,ref=ghcr.io/go-gitea/gitea:buildcache-rootless

View File

@@ -7,18 +7,17 @@ concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
permissions:
contents: read
jobs: jobs:
files-changed: files-changed:
uses: ./.github/workflows/files-changed.yml uses: ./.github/workflows/files-changed.yml
permissions:
contents: read
lint-backend: lint-backend:
if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true' if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true'
needs: files-changed needs: files-changed
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions:
contents: read
steps: steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0 - uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
@@ -35,93 +34,40 @@ jobs:
env: env:
TAGS: bindata sqlite sqlite_unlock_notify TAGS: bindata sqlite sqlite_unlock_notify
lint-templates: lint-on-demand:
if: needs.files-changed.outputs.templates == 'true'
needs: files-changed needs: files-changed
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
- run: uv python install 3.14
- uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: 24
cache: pnpm
cache-dependency-path: pnpm-lock.yaml
- run: make deps-py
- run: make deps-frontend
- run: make lint-templates
lint-yaml:
if: needs.files-changed.outputs.yaml == 'true'
needs: files-changed
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
- run: uv python install 3.14
- run: make deps-py
- run: make lint-yaml
lint-json:
if: needs.files-changed.outputs.json == 'true'
needs: files-changed
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: 24
cache: pnpm
cache-dependency-path: pnpm-lock.yaml
- run: make deps-frontend
- run: make lint-json
lint-swagger:
if: needs.files-changed.outputs.swagger == 'true'
needs: files-changed
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: 24
cache: pnpm
cache-dependency-path: pnpm-lock.yaml
- run: make deps-frontend
- run: make lint-swagger
lint-spell:
if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.frontend == 'true' || needs.files-changed.outputs.actions == 'true' || needs.files-changed.outputs.docs == 'true' || needs.files-changed.outputs.templates == 'true'
needs: files-changed
runs-on: ubuntu-latest
permissions:
contents: read
steps: steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0 - uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
with: with:
go-version-file: go.mod go-version-file: go.mod
check-latest: true check-latest: true
cache: false
- uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: 24
cache: pnpm
cache-dependency-path: pnpm-lock.yaml
- run: make lint-spell - run: make lint-spell
- if: needs.files-changed.outputs.templates == 'true' || needs.files-changed.outputs.yaml == 'true'
uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
- if: needs.files-changed.outputs.templates == 'true' || needs.files-changed.outputs.yaml == 'true'
run: uv python install 3.14 && make deps-py lint-templates lint-yaml
- if: needs.files-changed.outputs.docs == 'true' || needs.files-changed.outputs.swagger == 'true' || needs.files-changed.outputs.json == 'true'
run: make deps-frontend lint-md lint-swagger lint-json
- if: needs.files-changed.outputs.actions == 'true'
run: make lint-actions
lint-go-windows: lint-go-windows:
if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true' if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true'
needs: files-changed needs: files-changed
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions:
contents: read
steps: steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0 - uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
@@ -144,8 +90,6 @@ jobs:
if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true' if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true'
needs: files-changed needs: files-changed
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions:
contents: read
steps: steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0 - uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
@@ -166,8 +110,6 @@ jobs:
if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true' if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true'
needs: files-changed needs: files-changed
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions:
contents: read
steps: steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0 - uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
@@ -186,8 +128,6 @@ jobs:
if: needs.files-changed.outputs.frontend == 'true' || needs.files-changed.outputs.actions == 'true' if: needs.files-changed.outputs.frontend == 'true' || needs.files-changed.outputs.actions == 'true'
needs: files-changed needs: files-changed
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions:
contents: read
steps: steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0 - uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
@@ -206,8 +146,6 @@ jobs:
if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true' if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true'
needs: files-changed needs: files-changed
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions:
contents: read
steps: steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0 - uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
@@ -217,13 +155,12 @@ jobs:
cache: false cache: false
- uses: ./.github/actions/go-cache - uses: ./.github/actions/go-cache
with: with:
cache-name: backend cache-name: compliance-backend
# no frontend build here as backend should be able to build - run: make deps-backend generate-go
# even without any frontend files # no frontend build here as backend should be able to build, even without any frontend files
- run: make deps-backend # CGO is not used when cross-compile, so these steps also test if the code is compatible with CGO disabled
- run: go build -o gitea_no_gcc # test if build succeeds without the sqlite tag
- name: build-backend-arm64 - name: build-backend-arm64
run: make backend # test cross compile run: go build -o gitea_linux_arm64
env: env:
GOOS: linux GOOS: linux
GOARCH: arm64 GOARCH: arm64
@@ -235,38 +172,7 @@ jobs:
GOARCH: amd64 GOARCH: amd64
TAGS: bindata gogit TAGS: bindata gogit
- name: build-backend-386 - name: build-backend-386
run: go build -o gitea_linux_386 # test if compatible with 32 bit run: go build -o gitea_linux_386
env: env:
GOOS: linux GOOS: linux
GOARCH: 386 GOARCH: 386
docs:
if: needs.files-changed.outputs.docs == 'true' || needs.files-changed.outputs.actions == 'true'
needs: files-changed
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: 24
cache: pnpm
cache-dependency-path: pnpm-lock.yaml
- run: make deps-frontend
- run: make lint-md
actions:
if: needs.files-changed.outputs.actions == 'true' || needs.files-changed.outputs.actions == 'true'
needs: files-changed
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
with:
go-version-file: go.mod
check-latest: true
- run: make lint-actions

View File

@@ -7,18 +7,17 @@ concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
permissions:
contents: read
jobs: jobs:
files-changed: files-changed:
uses: ./.github/workflows/files-changed.yml uses: ./.github/workflows/files-changed.yml
permissions:
contents: read
test-pgsql: test-pgsql:
if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true' if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true'
needs: files-changed needs: files-changed
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions:
contents: read
services: services:
pgsql: pgsql:
image: postgres:14 image: postgres:14
@@ -70,8 +69,6 @@ jobs:
if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true' if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true'
needs: files-changed needs: files-changed
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions:
contents: read
steps: steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0 - uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
@@ -103,13 +100,12 @@ jobs:
if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true' if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true'
needs: files-changed needs: files-changed
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions:
contents: read
services: services:
elasticsearch: elasticsearch:
image: elasticsearch:7.5.0 image: docker.elastic.co/elasticsearch/elasticsearch:8.19.14
env: env:
discovery.type: single-node discovery.type: single-node
xpack.security.enabled: false
ports: ports:
- "9200:9200" - "9200:9200"
meilisearch: meilisearch:
@@ -173,8 +169,6 @@ jobs:
if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true' if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true'
needs: files-changed needs: files-changed
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions:
contents: read
services: services:
mysql: mysql:
# the bitnami mysql image has more options than the official one, it's easier to customize # the bitnami mysql image has more options than the official one, it's easier to customize
@@ -187,9 +181,10 @@ jobs:
options: >- options: >-
--mount type=tmpfs,destination=/bitnami/mysql/data --mount type=tmpfs,destination=/bitnami/mysql/data
elasticsearch: elasticsearch:
image: elasticsearch:7.5.0 image: docker.elastic.co/elasticsearch/elasticsearch:8.19.14
env: env:
discovery.type: single-node discovery.type: single-node
xpack.security.enabled: false
ports: ports:
- "9200:9200" - "9200:9200"
smtpimap: smtpimap:
@@ -227,8 +222,6 @@ jobs:
if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true' if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true'
needs: files-changed needs: files-changed
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions:
contents: read
services: services:
mssql: mssql:
image: mcr.microsoft.com/mssql/server:2019-latest image: mcr.microsoft.com/mssql/server:2019-latest

View File

@@ -13,27 +13,35 @@ permissions:
jobs: jobs:
files-changed: files-changed:
uses: ./.github/workflows/files-changed.yml uses: ./.github/workflows/files-changed.yml
permissions:
contents: read
# dryrun build is slow, so run them in parallel per-platform # QEMU-based build is slow (40-50 minutes), so run arm64 and riscv64 when dockerfile changes.
# Run amd64 when any docker-related files change, which is fast (4 minutes).
container-amd64: container-amd64:
if: needs.files-changed.outputs.docker == 'true' if: needs.files-changed.outputs.docker == 'true'
needs: [files-changed] needs: [files-changed]
uses: ./.github/workflows/part-docker-dryrun.yml runs-on: ubuntu-latest
with: steps:
platform: linux/amd64 - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: ./.github/actions/docker-dryrun
with:
platform: linux/amd64
container-arm64: container-arm64:
if: needs.files-changed.outputs.docker == 'true' if: needs.files-changed.outputs.dockerfile == 'true'
needs: [files-changed] needs: [files-changed]
uses: ./.github/workflows/part-docker-dryrun.yml runs-on: ubuntu-latest
with: steps:
platform: linux/arm64 - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: ./.github/actions/docker-dryrun
with:
platform: linux/arm64
container-riscv64: container-riscv64:
if: needs.files-changed.outputs.docker == 'true' if: needs.files-changed.outputs.dockerfile == 'true'
needs: [files-changed] needs: [files-changed]
uses: ./.github/workflows/part-docker-dryrun.yml runs-on: ubuntu-latest
with: steps:
platform: linux/riscv64 - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: ./.github/actions/docker-dryrun
with:
platform: linux/riscv64

View File

@@ -7,18 +7,17 @@ concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true cancel-in-progress: true
permissions:
contents: read
jobs: jobs:
files-changed: files-changed:
uses: ./.github/workflows/files-changed.yml uses: ./.github/workflows/files-changed.yml
permissions:
contents: read
test-e2e: test-e2e:
if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.frontend == 'true' || needs.files-changed.outputs.e2e == 'true' if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.frontend == 'true' || needs.files-changed.outputs.e2e == 'true'
needs: files-changed needs: files-changed
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions:
contents: read
steps: steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0 - uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0

1
.gitignore vendored
View File

@@ -65,6 +65,7 @@ cpu.out
/indexers /indexers
/log /log
/public/assets/img/avatar /public/assets/img/avatar
/tests/e2e-output
/tests/integration/gitea-integration-* /tests/integration/gitea-integration-*
/tests/integration/indexers-* /tests/integration/indexers-*
/tests/*.ini /tests/*.ini

View File

@@ -164,7 +164,7 @@ TEST_PGSQL_PASSWORD ?= postgres
TEST_PGSQL_SCHEMA ?= gtestschema TEST_PGSQL_SCHEMA ?= gtestschema
TEST_MINIO_ENDPOINT ?= minio:9000 TEST_MINIO_ENDPOINT ?= minio:9000
TEST_MSSQL_HOST ?= mssql:1433 TEST_MSSQL_HOST ?= mssql:1433
TEST_MSSQL_DBNAME ?= gitea TEST_MSSQL_DBNAME ?= testgitea
TEST_MSSQL_USERNAME ?= sa TEST_MSSQL_USERNAME ?= sa
TEST_MSSQL_PASSWORD ?= MwantsaSecurePassword1 TEST_MSSQL_PASSWORD ?= MwantsaSecurePassword1
@@ -274,7 +274,7 @@ checks-frontend: lockfile-check svg-check ## check frontend files
checks-backend: tidy-check swagger-check openapi3-check fmt-check swagger-validate security-check ## check backend files checks-backend: tidy-check swagger-check openapi3-check fmt-check swagger-validate security-check ## check backend files
.PHONY: lint .PHONY: lint
lint: lint-frontend lint-backend lint-spell ## lint everything lint: lint-frontend lint-backend lint-templates lint-swagger lint-spell lint-md lint-actions lint-json lint-yaml ## lint everything
.PHONY: lint-fix .PHONY: lint-fix
lint-fix: lint-frontend-fix lint-backend-fix lint-spell-fix ## lint everything and fix issues lint-fix: lint-frontend-fix lint-backend-fix lint-spell-fix ## lint everything and fix issues
@@ -478,7 +478,7 @@ playwright: deps-frontend
@pnpm exec playwright install $(if $(GITHUB_ACTIONS),,--with-deps) chromium firefox $(PLAYWRIGHT_FLAGS) @pnpm exec playwright install $(if $(GITHUB_ACTIONS),,--with-deps) chromium firefox $(PLAYWRIGHT_FLAGS)
.PHONY: test-e2e .PHONY: test-e2e
test-e2e: playwright backend test-e2e: playwright frontend backend
@EXECUTABLE=$(EXECUTABLE) ./tools/test-e2e.sh $(GITEA_TEST_E2E_FLAGS) @EXECUTABLE=$(EXECUTABLE) ./tools/test-e2e.sh $(GITEA_TEST_E2E_FLAGS)
.PHONY: build .PHONY: build

View File

@@ -1004,16 +1004,6 @@
"path": "github.com/olekukonko/tablewriter/LICENSE.md", "path": "github.com/olekukonko/tablewriter/LICENSE.md",
"licenseText": "Copyright (C) 2014 by Oleku Konko\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in\nall copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\nTHE SOFTWARE.\n" "licenseText": "Copyright (C) 2014 by Oleku Konko\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in\nall copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\nTHE SOFTWARE.\n"
}, },
{
"name": "github.com/olivere/elastic/v7",
"path": "github.com/olivere/elastic/v7/LICENSE",
"licenseText": "The MIT License (MIT)\nCopyright © 2012-2015 Oliver Eilhard\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the “Software”), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included\nin all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING\nFROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS\nIN THE SOFTWARE.\n"
},
{
"name": "github.com/olivere/elastic/v7/uritemplates",
"path": "github.com/olivere/elastic/v7/uritemplates/LICENSE",
"licenseText": "Copyright (c) 2013 Joshua Tacoma\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of\nthis software and associated documentation files (the \"Software\"), to deal in\nthe Software without restriction, including without limitation the rights to\nuse, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of\nthe Software, and to permit persons to whom the Software is furnished to do so,\nsubject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS\nFOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR\nCOPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER\nIN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN\nCONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n"
},
{ {
"name": "github.com/opencontainers/go-digest", "name": "github.com/opencontainers/go-digest",
"path": "github.com/opencontainers/go-digest/LICENSE", "path": "github.com/opencontainers/go-digest/LICENSE",

View File

@@ -203,8 +203,8 @@ func runDump(ctx context.Context, cmd *cli.Command) error {
} }
}() }()
targetDBType := cmd.String("database") targetDBType := setting.DatabaseType(cmd.String("database"))
if len(targetDBType) > 0 && targetDBType != setting.Database.Type.String() { if targetDBType != "" && targetDBType != setting.Database.Type {
log.Info("Dumping database %s => %s...", setting.Database.Type, targetDBType) log.Info("Dumping database %s => %s...", setting.Database.Type, targetDBType)
} else { } else {
log.Info("Dumping database...") log.Info("Dumping database...")

View File

@@ -1524,7 +1524,7 @@ LEVEL = Info
;; Issue Indexer settings ;; Issue Indexer settings
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; ;;
;; Issue indexer type, currently support: bleve, db, elasticsearch or meilisearch default is bleve ;; Issue indexer type, currently support: bleve, db, elasticsearch (also compatible with OpenSearch) or meilisearch default is bleve
;ISSUE_INDEXER_TYPE = bleve ;ISSUE_INDEXER_TYPE = bleve
;; ;;
;; Issue indexer storage path, available when ISSUE_INDEXER_TYPE is bleve ;; Issue indexer storage path, available when ISSUE_INDEXER_TYPE is bleve
@@ -1551,7 +1551,7 @@ LEVEL = Info
;; If empty then it defaults to `sources` only, as if you'd like to disable fully please see REPO_INDEXER_ENABLED. ;; If empty then it defaults to `sources` only, as if you'd like to disable fully please see REPO_INDEXER_ENABLED.
;REPO_INDEXER_REPO_TYPES = sources,forks,mirrors,templates ;REPO_INDEXER_REPO_TYPES = sources,forks,mirrors,templates
;; ;;
;; Code search engine type, could be `bleve` or `elasticsearch`. ;; Code search engine type, could be `bleve` or `elasticsearch` (also compatible with OpenSearch).
;REPO_INDEXER_TYPE = bleve ;REPO_INDEXER_TYPE = bleve
;; ;;
;; Index file used for code search. available when `REPO_INDEXER_TYPE` is bleve ;; Index file used for code search. available when `REPO_INDEXER_TYPE` is bleve

3
go.mod
View File

@@ -87,7 +87,6 @@ require (
github.com/msteinert/pam/v2 v2.1.0 github.com/msteinert/pam/v2 v2.1.0
github.com/nektos/act v0.2.63 github.com/nektos/act v0.2.63
github.com/niklasfasching/go-org v1.9.1 github.com/niklasfasching/go-org v1.9.1
github.com/olivere/elastic/v7 v7.0.32
github.com/opencontainers/go-digest v1.0.0 github.com/opencontainers/go-digest v1.0.0
github.com/opencontainers/image-spec v1.1.1 github.com/opencontainers/image-spec v1.1.1
github.com/pquerna/otp v1.5.0 github.com/pquerna/otp v1.5.0
@@ -222,7 +221,7 @@ require (
github.com/klauspost/crc32 v1.3.0 // indirect github.com/klauspost/crc32 v1.3.0 // indirect
github.com/klauspost/pgzip v1.2.6 // indirect github.com/klauspost/pgzip v1.2.6 // indirect
github.com/libdns/libdns v1.1.1 // indirect github.com/libdns/libdns v1.1.1 // indirect
github.com/mailru/easyjson v0.9.2 // indirect github.com/mailru/easyjson v0.7.7 // indirect
github.com/markbates/going v1.0.3 // indirect github.com/markbates/going v1.0.3 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-runewidth v0.0.21 // indirect github.com/mattn/go-runewidth v0.0.21 // indirect

10
go.sum
View File

@@ -267,8 +267,6 @@ github.com/fatih/color v1.19.0 h1:Zp3PiM21/9Ld6FzSKyL5c/BULoe/ONr9KlbYVOfG8+w=
github.com/fatih/color v1.19.0/go.mod h1:zNk67I0ZUT1bEGsSGyCZYZNrHuTkJJB+r6Q9VuMi0LE= github.com/fatih/color v1.19.0/go.mod h1:zNk67I0ZUT1bEGsSGyCZYZNrHuTkJJB+r6Q9VuMi0LE=
github.com/felixge/fgprof v0.9.5 h1:8+vR6yu2vvSKn08urWyEuxx75NWPEvybbkBirEpsbVY= github.com/felixge/fgprof v0.9.5 h1:8+vR6yu2vvSKn08urWyEuxx75NWPEvybbkBirEpsbVY=
github.com/felixge/fgprof v0.9.5/go.mod h1:yKl+ERSa++RYOs32d8K6WEXCB4uXdLls4ZaZPpayhMM= github.com/felixge/fgprof v0.9.5/go.mod h1:yKl+ERSa++RYOs32d8K6WEXCB4uXdLls4ZaZPpayhMM=
github.com/fortytw2/leaktest v1.3.0 h1:u8491cBMTQ8ft8aeV+adlcytMZylmA5nnwwkRZjI8vw=
github.com/fortytw2/leaktest v1.3.0/go.mod h1:jDsjWgpAGjm2CA7WthBh/CdZYEPF31XHquHwclZch5g=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ= github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k= github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
@@ -507,9 +505,8 @@ github.com/lib/pq v1.12.3/go.mod h1:/p+8NSbOcwzAEI7wiMXFlgydTwcgTr3OSKMsD2BitpA=
github.com/libdns/libdns v1.1.1 h1:wPrHrXILoSHKWJKGd0EiAVmiJbFShguILTg9leS/P/U= github.com/libdns/libdns v1.1.1 h1:wPrHrXILoSHKWJKGd0EiAVmiJbFShguILTg9leS/P/U=
github.com/libdns/libdns v1.1.1/go.mod h1:4Bj9+5CQiNMVGf87wjX4CY3HQJypUHRuLvlsfsZqLWQ= github.com/libdns/libdns v1.1.1/go.mod h1:4Bj9+5CQiNMVGf87wjX4CY3HQJypUHRuLvlsfsZqLWQ=
github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0=
github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
github.com/mailru/easyjson v0.9.2 h1:dX8U45hQsZpxd80nLvDGihsQ/OxlvTkVUXH2r/8cb2M=
github.com/mailru/easyjson v0.9.2/go.mod h1:1+xMtQp2MRNVL/V1bOzuP3aP8VNwRW55fQUto+XFtTU=
github.com/markbates/going v1.0.3 h1:mY45T5TvW+Xz5A6jY7lf4+NLg9D8+iuStIHyR7M8qsE= github.com/markbates/going v1.0.3 h1:mY45T5TvW+Xz5A6jY7lf4+NLg9D8+iuStIHyR7M8qsE=
github.com/markbates/going v1.0.3/go.mod h1:fQiT6v6yQar9UD6bd/D4Z5Afbk9J6BBVBtLiyY4gp2o= github.com/markbates/going v1.0.3/go.mod h1:fQiT6v6yQar9UD6bd/D4Z5Afbk9J6BBVBtLiyY4gp2o=
github.com/markbates/goth v1.82.0 h1:8j/c34AjBSTNzO7zTsOyP5IYCQCMBTRBHAbBt/PI0bQ= github.com/markbates/goth v1.82.0 h1:8j/c34AjBSTNzO7zTsOyP5IYCQCMBTRBHAbBt/PI0bQ=
@@ -585,8 +582,6 @@ github.com/olekukonko/ll v0.1.8 h1:ysHCJRGHYKzmBSdz9w5AySztx7lG8SQY+naTGYUbsz8=
github.com/olekukonko/ll v0.1.8/go.mod h1:RPRC6UcscfFZgjo1nulkfMH5IM0QAYim0LfnMvUuozw= github.com/olekukonko/ll v0.1.8/go.mod h1:RPRC6UcscfFZgjo1nulkfMH5IM0QAYim0LfnMvUuozw=
github.com/olekukonko/tablewriter v1.1.4 h1:ORUMI3dXbMnRlRggJX3+q7OzQFDdvgbN9nVWj1drm6I= github.com/olekukonko/tablewriter v1.1.4 h1:ORUMI3dXbMnRlRggJX3+q7OzQFDdvgbN9nVWj1drm6I=
github.com/olekukonko/tablewriter v1.1.4/go.mod h1:+kedxuyTtgoZLwif3P1Em4hARJs+mVnzKxmsCL/C5RY= github.com/olekukonko/tablewriter v1.1.4/go.mod h1:+kedxuyTtgoZLwif3P1Em4hARJs+mVnzKxmsCL/C5RY=
github.com/olivere/elastic/v7 v7.0.32 h1:R7CXvbu8Eq+WlsLgxmKVKPox0oOwAE/2T9Si5BnvK6E=
github.com/olivere/elastic/v7 v7.0.32/go.mod h1:c7PVmLe3Fxq77PIfY/bZmxY/TAamBhCzZ8xDOE09a9k=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk= github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk=
@@ -667,9 +662,8 @@ github.com/sirupsen/logrus v1.9.4 h1:TsZE7l11zFCLZnZ+teH4Umoq5BhEIfIzfRDZ1Uzql2w
github.com/sirupsen/logrus v1.9.4/go.mod h1:ftWc9WdOfJ0a92nsE2jF5u5ZwH8Bv2zdeOC42RjbV2g= github.com/sirupsen/logrus v1.9.4/go.mod h1:ftWc9WdOfJ0a92nsE2jF5u5ZwH8Bv2zdeOC42RjbV2g=
github.com/skeema/knownhosts v1.3.2 h1:EDL9mgf4NzwMXCTfaxSD/o/a5fxDw/xL9nkU28JjdBg= github.com/skeema/knownhosts v1.3.2 h1:EDL9mgf4NzwMXCTfaxSD/o/a5fxDw/xL9nkU28JjdBg=
github.com/skeema/knownhosts v1.3.2/go.mod h1:bEg3iQAuw+jyiw+484wwFJoKSLwcfd7fqRy+N0QTiow= github.com/skeema/knownhosts v1.3.2/go.mod h1:bEg3iQAuw+jyiw+484wwFJoKSLwcfd7fqRy+N0QTiow=
github.com/smartystreets/assertions v0.0.0-20190116191733-b6c0e53d7304 h1:Jpy1PXuP99tXNrhbq2BaPz9B+jNAvH1JPQQpG/9GCXY=
github.com/smartystreets/assertions v0.0.0-20190116191733-b6c0e53d7304/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc= github.com/smartystreets/assertions v0.0.0-20190116191733-b6c0e53d7304/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
github.com/smartystreets/assertions v1.1.1 h1:T/YLemO5Yp7KPzS+lVtu+WsHn8yoSwTfItdAd1r3cck=
github.com/smartystreets/assertions v1.1.1/go.mod h1:tcbTF8ujkAEcZ8TElKY+i30BzYlVhC/LOxJk7iOWnoo=
github.com/smartystreets/goconvey v0.0.0-20181108003508-044398e4856c/go.mod h1:XDJAKZRPZ1CvBcN2aX5YOUTYGHki24fSF0Iv48Ibg0s= github.com/smartystreets/goconvey v0.0.0-20181108003508-044398e4856c/go.mod h1:XDJAKZRPZ1CvBcN2aX5YOUTYGHki24fSF0Iv48Ibg0s=
github.com/smartystreets/goconvey v0.0.0-20190731233626-505e41936337 h1:WN9BUFbdyOsSH/XohnWpXOlq9NBD5sGAB2FciQMUEe8= github.com/smartystreets/goconvey v0.0.0-20190731233626-505e41936337 h1:WN9BUFbdyOsSH/XohnWpXOlq9NBD5sGAB2FciQMUEe8=
github.com/smartystreets/goconvey v0.0.0-20190731233626-505e41936337/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA= github.com/smartystreets/goconvey v0.0.0-20190731233626-505e41936337/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=

173
models/db/conn.go Normal file
View File

@@ -0,0 +1,173 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package db
import (
"errors"
"fmt"
"net"
"net/url"
"os"
"path/filepath"
"strings"
"code.gitea.io/gitea/modules/setting"
"code.gitea.io/gitea/modules/util"
)
type ConnOptions struct {
Type setting.DatabaseType
Host string
Database string
User string
Passwd string
Schema string
SSLMode string
SQLitePath string
SQLiteBusyTimeout int
SQLiteJournalMode string
}
type SQLiteConnStrOptions struct {
FilePath string
BusyTimeout int
JournalMode string
}
func GlobalConnOptions() ConnOptions {
return ConnOptions{
Type: setting.Database.Type,
Host: setting.Database.Host,
Database: setting.Database.Name,
User: setting.Database.User,
Passwd: setting.Database.Passwd,
Schema: setting.Database.Schema,
SSLMode: setting.Database.SSLMode,
SQLitePath: setting.Database.Path,
SQLiteBusyTimeout: setting.Database.SQLiteBusyTimeout,
SQLiteJournalMode: setting.Database.SQLiteJournalMode,
}
}
const sqlDriverPostgresSchema = "postgresschema"
var makeSQLiteConnStr = func(opts SQLiteConnStrOptions) (string, string, error) {
return "", "", errors.New(`this Gitea binary was not built with SQLite3 support, get an official release or rebuild with: -tags sqlite,sqlite_unlock_notify`)
}
func ConnStrDefaultDatabase(opts ConnOptions) (string, string, error) {
opts.Database, opts.Schema = "", ""
return ConnStr(opts)
}
func ConnStr(opts ConnOptions) (string, string, error) {
switch {
case opts.Type.IsMySQL():
// use unix socket or tcp socket
connType := util.Iif(strings.HasPrefix(opts.Host, "/"), "unix", "tcp")
// allow (Postgres-inspired) default value to work in MySQL
tls := util.Iif(opts.SSLMode == "disable", "false", opts.SSLMode)
// in case the database name is a partial connection string which contains "?" parameters
paramSep := util.Iif(strings.Contains(opts.Database, "?"), "&", "?")
connStr := fmt.Sprintf("%s:%s@%s(%s)/%s%sparseTime=true&tls=%s", opts.User, opts.Passwd, connType, opts.Host, opts.Database, paramSep, tls)
return "mysql", connStr, nil
case opts.Type.IsPostgreSQL():
connStr := makePgSQLConnStr(opts.Host, opts.User, opts.Passwd, opts.Database, opts.SSLMode)
driver := util.Iif(opts.Schema == "", "postgres", sqlDriverPostgresSchema)
registerPostgresSchemaDriver()
return driver, connStr, nil
case opts.Type.IsMSSQL():
host, port := parseMSSQLHostPort(opts.Host)
connStr := fmt.Sprintf("server=%s; port=%s; user id=%s; password=%s;", host, port, opts.User, opts.Passwd)
if opts.Database != "" {
connStr += "; database=" + opts.Database
}
return "mssql", connStr, nil
case opts.Type.IsSQLite3():
if opts.SQLitePath == "" {
return "", "", errors.New("sqlite3 database path cannot be empty")
}
if err := os.MkdirAll(filepath.Dir(opts.SQLitePath), os.ModePerm); err != nil {
return "", "", fmt.Errorf("failed to create directories: %w", err)
}
return makeSQLiteConnStr(SQLiteConnStrOptions{
FilePath: opts.SQLitePath,
JournalMode: opts.SQLiteJournalMode,
BusyTimeout: opts.SQLiteBusyTimeout,
})
}
return "", "", fmt.Errorf("unknown database type: %s", opts.Type)
}
// parsePgSQLHostPort parses given input in various forms defined in
// https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-CONNSTRING
// and returns proper host and port number.
func parsePgSQLHostPort(info string) (host, port string) {
if h, p, err := net.SplitHostPort(info); err == nil {
host, port = h, p
} else {
// treat the "info" as "host", if it's an IPv6 address, remove the wrapper
host = info
if strings.HasPrefix(host, "[") && strings.HasSuffix(host, "]") {
host = host[1 : len(host)-1]
}
}
// set fallback values
if host == "" {
host = "127.0.0.1"
}
if port == "" {
port = "5432"
}
return host, port
}
func makePgSQLConnStr(dbHost, dbUser, dbPasswd, dbName, dbsslMode string) (connStr string) {
dbName, dbParam, _ := strings.Cut(dbName, "?")
host, port := parsePgSQLHostPort(dbHost)
connURL := url.URL{
Scheme: "postgres",
User: url.UserPassword(dbUser, dbPasswd),
Host: net.JoinHostPort(host, port),
Path: dbName,
OmitHost: false,
RawQuery: dbParam,
}
query := connURL.Query()
if strings.HasPrefix(host, "/") { // looks like a unix socket
query.Add("host", host)
connURL.Host = ":" + port
}
query.Set("sslmode", dbsslMode)
connURL.RawQuery = query.Encode()
return connURL.String()
}
// parseMSSQLHostPort splits the host into host and port
func parseMSSQLHostPort(info string) (string, string) {
// the default port "0" might be related to MSSQL's dynamic port, maybe it should be double-confirmed in the future
host, port := "127.0.0.1", "0"
if strings.Contains(info, ":") {
host = strings.Split(info, ":")[0]
port = strings.Split(info, ":")[1]
} else if strings.Contains(info, ",") {
host = strings.Split(info, ",")[0]
port = strings.TrimSpace(strings.Split(info, ",")[1])
} else if len(info) > 0 {
host = info
}
if host == "" {
host = "127.0.0.1"
}
if port == "" {
port = "0"
}
return host, port
}

View File

@@ -1,7 +1,7 @@
// Copyright 2019 The Gitea Authors. All rights reserved. // Copyright 2019 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
package setting package db
import ( import (
"testing" "testing"
@@ -9,7 +9,7 @@ import (
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
) )
func Test_parsePostgreSQLHostPort(t *testing.T) { func TestParsePgSQLHostPort(t *testing.T) {
tests := map[string]struct { tests := map[string]struct {
HostPort string HostPort string
Host string Host string
@@ -49,14 +49,14 @@ func Test_parsePostgreSQLHostPort(t *testing.T) {
for k, test := range tests { for k, test := range tests {
t.Run(k, func(t *testing.T) { t.Run(k, func(t *testing.T) {
t.Log(test.HostPort) t.Log(test.HostPort)
host, port := parsePostgreSQLHostPort(test.HostPort) host, port := parsePgSQLHostPort(test.HostPort)
assert.Equal(t, test.Host, host) assert.Equal(t, test.Host, host)
assert.Equal(t, test.Port, port) assert.Equal(t, test.Port, port)
}) })
} }
} }
func Test_getPostgreSQLConnectionString(t *testing.T) { func TestMakePgSQLConnStr(t *testing.T) {
tests := []struct { tests := []struct {
Host string Host string
User string User string
@@ -103,7 +103,7 @@ func Test_getPostgreSQLConnectionString(t *testing.T) {
} }
for _, test := range tests { for _, test := range tests {
connStr := getPostgreSQLConnectionString(test.Host, test.User, test.Passwd, test.Name, test.SSLMode) connStr := makePgSQLConnStr(test.Host, test.User, test.Passwd, test.Name, test.SSLMode)
assert.Equal(t, test.Output, connStr) assert.Equal(t, test.Output, connStr)
} }
} }

View File

@@ -18,8 +18,8 @@ var registerOnce sync.Once
func registerPostgresSchemaDriver() { func registerPostgresSchemaDriver() {
registerOnce.Do(func() { registerOnce.Do(func() {
sql.Register("postgresschema", &postgresSchemaDriver{}) sql.Register(sqlDriverPostgresSchema, &postgresSchemaDriver{})
dialects.RegisterDriver("postgresschema", dialects.QueryDriver("postgres")) dialects.RegisterDriver(sqlDriverPostgresSchema, dialects.QueryDriver("postgres"))
}) })
} }

View File

@@ -0,0 +1,34 @@
//go:build sqlite
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package db
import (
"fmt"
"strconv"
"strings"
"code.gitea.io/gitea/modules/setting"
_ "github.com/mattn/go-sqlite3"
)
func init() {
setting.SupportedDatabaseTypes = append(setting.SupportedDatabaseTypes, "sqlite3")
makeSQLiteConnStr = makeSQLiteConnStrMattnCGO
}
func makeSQLiteConnStrMattnCGO(opts SQLiteConnStrOptions) (string, string, error) {
var params []string
params = append(params, "cache=shared")
params = append(params, "mode=rwc")
params = append(params, "_busy_timeout="+strconv.Itoa(opts.BusyTimeout))
params = append(params, "_txlock=immediate")
if opts.JournalMode != "" {
params = append(params, "_journal_mode="+opts.JournalMode)
}
connStr := fmt.Sprintf("file:%s?%s", opts.FilePath, strings.Join(params, "&"))
return "sqlite3", connStr, nil
}

View File

@@ -3,10 +3,14 @@
package db package db
import "xorm.io/xorm/schemas" import (
"code.gitea.io/gitea/modules/setting"
"xorm.io/xorm/schemas"
)
// DumpDatabase dumps all data from database according the special database SQL syntax to file system. // DumpDatabase dumps all data from database according the special database SQL syntax to file system.
func DumpDatabase(filePath, dbType string) error { func DumpDatabase(filePath string, dbType setting.DatabaseType) error {
var tbs []*schemas.Table var tbs []*schemas.Table
for _, t := range registeredModels { for _, t := range registeredModels {
t, err := xormEngine.TableInfo(t) t, err := xormEngine.TableInfo(t)

View File

@@ -6,7 +6,6 @@ package db
import ( import (
"context" "context"
"fmt" "fmt"
"strings"
"code.gitea.io/gitea/modules/log" "code.gitea.io/gitea/modules/log"
"code.gitea.io/gitea/modules/setting" "code.gitea.io/gitea/modules/setting"
@@ -24,31 +23,23 @@ func init() {
// newXORMEngine returns a new XORM engine from the configuration // newXORMEngine returns a new XORM engine from the configuration
func newXORMEngine() (*xorm.Engine, error) { func newXORMEngine() (*xorm.Engine, error) {
connStr, err := setting.DBConnStr() connOpts := GlobalConnOptions()
driver, connStr, err := ConnStr(connOpts)
if err != nil { if err != nil {
return nil, err return nil, err
} }
var engine *xorm.Engine engine, err := xorm.NewEngine(driver, connStr)
if setting.Database.Type.IsPostgreSQL() && len(setting.Database.Schema) > 0 {
// OK whilst we sort out our schema issues - create a schema aware postgres
registerPostgresSchemaDriver()
engine, err = xorm.NewEngine("postgresschema", connStr)
} else {
engine, err = xorm.NewEngine(setting.Database.Type.String(), connStr)
}
if err != nil { if err != nil {
return nil, err return nil, err
} }
switch setting.Database.Type { switch {
case "mysql": case connOpts.Type.IsMySQL():
engine.Dialect().SetParams(map[string]string{"rowFormat": "DYNAMIC"}) engine.Dialect().SetParams(map[string]string{"rowFormat": "DYNAMIC"})
case "mssql": case connOpts.Type.IsMSSQL():
engine.Dialect().SetParams(map[string]string{"DEFAULT_VARCHAR": "nvarchar"}) engine.Dialect().SetParams(map[string]string{"DEFAULT_VARCHAR": "nvarchar"})
} }
engine.SetSchema(setting.Database.Schema) engine.SetSchema(connOpts.Schema)
return engine, nil return engine, nil
} }
@@ -56,10 +47,7 @@ func newXORMEngine() (*xorm.Engine, error) {
func InitEngine(ctx context.Context) error { func InitEngine(ctx context.Context) error {
xe, err := newXORMEngine() xe, err := newXORMEngine()
if err != nil { if err != nil {
if strings.Contains(err.Error(), "SQLite3 support") { return fmt.Errorf("failed to init database engine: %w", err)
return fmt.Errorf("sqlite3 requires: -tags sqlite,sqlite_unlock_notify\n%w", err)
}
return fmt.Errorf("failed to connect to database: %w", err)
} }
xe.SetMapper(names.GonicMapper{}) xe.SetMapper(names.GonicMapper{})

View File

@@ -30,7 +30,7 @@ func TestDumpDatabase(t *testing.T) {
assert.NoError(t, db.GetEngine(t.Context()).Sync(new(Version))) assert.NoError(t, db.GetEngine(t.Context()).Sync(new(Version)))
for _, dbType := range setting.SupportedDatabaseTypes { for _, dbType := range setting.SupportedDatabaseTypes {
assert.NoError(t, db.DumpDatabase(filepath.Join(dir, dbType+".sql"), dbType)) assert.NoError(t, db.DumpDatabase(filepath.Join(dir, dbType+".sql"), setting.DatabaseType(dbType)))
} }
} }

View File

@@ -59,17 +59,18 @@ type Issue struct {
PosterID int64 `xorm:"INDEX"` PosterID int64 `xorm:"INDEX"`
Poster *user_model.User `xorm:"-"` Poster *user_model.User `xorm:"-"`
OriginalAuthor string OriginalAuthor string
OriginalAuthorID int64 `xorm:"index"` OriginalAuthorID int64 `xorm:"index"`
Title string `xorm:"name"` Title string `xorm:"name"`
Content string `xorm:"LONGTEXT"` Content string `xorm:"LONGTEXT"`
RenderedContent template.HTML `xorm:"-"` RenderedContent template.HTML `xorm:"-"`
ContentVersion int `xorm:"NOT NULL DEFAULT 0"` ContentVersion int `xorm:"NOT NULL DEFAULT 0"`
Labels []*Label `xorm:"-"` Labels []*Label `xorm:"-"`
isLabelsLoaded bool `xorm:"-"` isLabelsLoaded bool `xorm:"-"`
MilestoneID int64 `xorm:"INDEX"` MilestoneID int64 `xorm:"INDEX"`
Milestone *Milestone `xorm:"-"` Milestone *Milestone `xorm:"-"`
isMilestoneLoaded bool `xorm:"-"` isMilestoneLoaded bool `xorm:"-"`
Project *project_model.Project `xorm:"-"` Projects []*project_model.Project `xorm:"-"`
isProjectsLoaded bool `xorm:"-"`
Priority int Priority int
AssigneeID int64 `xorm:"-"` AssigneeID int64 `xorm:"-"`
Assignee *user_model.User `xorm:"-"` Assignee *user_model.User `xorm:"-"`
@@ -305,7 +306,7 @@ func (issue *Issue) LoadAttributes(ctx context.Context) (err error) {
return err return err
} }
if err = issue.LoadProject(ctx); err != nil { if err = issue.LoadProjects(ctx); err != nil {
return err return err
} }
@@ -355,6 +356,7 @@ func (issue *Issue) ResetAttributesLoaded() {
issue.isMilestoneLoaded = false issue.isMilestoneLoaded = false
issue.isAttachmentsLoaded = false issue.isAttachmentsLoaded = false
issue.isAssigneeLoaded = false issue.isAssigneeLoaded = false
issue.isProjectsLoaded = false
} }
// GetIsRead load the `IsRead` field of the issue // GetIsRead load the `IsRead` field of the issue

View File

@@ -185,7 +185,7 @@ func (issues IssueList) LoadMilestones(ctx context.Context) error {
func (issues IssueList) LoadProjects(ctx context.Context) error { func (issues IssueList) LoadProjects(ctx context.Context) error {
issueIDs := issues.getIssueIDs() issueIDs := issues.getIssueIDs()
projectMaps := make(map[int64]*project_model.Project, len(issues)) issueProjectMaps := make(map[int64][]*project_model.Project, len(issues))
left := len(issueIDs) left := len(issueIDs)
type projectWithIssueID struct { type projectWithIssueID struct {
@@ -202,19 +202,21 @@ func (issues IssueList) LoadProjects(ctx context.Context) error {
Select("project.*, project_issue.issue_id"). Select("project.*, project_issue.issue_id").
Join("INNER", "project_issue", "project.id = project_issue.project_id"). Join("INNER", "project_issue", "project.id = project_issue.project_id").
In("project_issue.issue_id", issueIDs[:limit]). In("project_issue.issue_id", issueIDs[:limit]).
OrderBy("project_issue.issue_id ASC, project.id ASC").
Find(&projects) Find(&projects)
if err != nil { if err != nil {
return err return err
} }
for _, project := range projects { for _, project := range projects {
projectMaps[project.IssueID] = project.Project issueProjectMaps[project.IssueID] = append(issueProjectMaps[project.IssueID], project.Project)
} }
left -= limit left -= limit
issueIDs = issueIDs[limit:] issueIDs = issueIDs[limit:]
} }
for _, issue := range issues { for _, issue := range issues {
issue.Project = projectMaps[issue.ID] issue.Projects = issueProjectMaps[issue.ID]
issue.isProjectsLoaded = true
} }
return nil return nil
} }

View File

@@ -65,10 +65,10 @@ func TestIssueList_LoadAttributes(t *testing.T) {
} }
if issue.ID == int64(1) { if issue.ID == int64(1) {
assert.Equal(t, int64(400), issue.TotalTrackedTime) assert.Equal(t, int64(400), issue.TotalTrackedTime)
assert.NotNil(t, issue.Project) assert.NotEmpty(t, issue.Projects)
assert.Equal(t, int64(1), issue.Project.ID) assert.Equal(t, int64(1), issue.Projects[0].ID)
} else { } else {
assert.Nil(t, issue.Project) assert.Empty(t, issue.Projects)
} }
} }
} }

View File

@@ -12,41 +12,38 @@ import (
"code.gitea.io/gitea/modules/util" "code.gitea.io/gitea/modules/util"
) )
// LoadProject load the project the issue was assigned to // LoadProjects loads all projects the issue is assigned to
func (issue *Issue) LoadProject(ctx context.Context) (err error) { func (issue *Issue) LoadProjects(ctx context.Context) (err error) {
if issue.Project == nil { if !issue.isProjectsLoaded {
var p project_model.Project err = db.GetEngine(ctx).Table("project").
has, err := db.GetEngine(ctx).Table("project").
Join("INNER", "project_issue", "project.id=project_issue.project_id"). Join("INNER", "project_issue", "project.id=project_issue.project_id").
Where("project_issue.issue_id = ?", issue.ID).Get(&p) Where("project_issue.issue_id = ?", issue.ID).
if err != nil { OrderBy("project.id ASC").
return err Find(&issue.Projects)
} else if has { if err == nil {
issue.Project = &p issue.isProjectsLoaded = true
} }
} }
return err return err
} }
func (issue *Issue) projectID(ctx context.Context) int64 { func (issue *Issue) projectIDs(ctx context.Context) (projectIDs []int64, _ error) {
var ip project_model.ProjectIssue err := db.GetEngine(ctx).Table("project_issue").Where("issue_id = ?", issue.ID).Cols("project_id").Find(&projectIDs)
has, err := db.GetEngine(ctx).Where("issue_id=?", issue.ID).Get(&ip) return projectIDs, err
if err != nil || !has {
return 0
}
return ip.ProjectID
} }
// ProjectColumnID return project column id if issue was assigned to one // ProjectColumnMap returns a map of project ID to column ID for this issue.
func (issue *Issue) ProjectColumnID(ctx context.Context) (int64, error) { func (issue *Issue) ProjectColumnMap(ctx context.Context) (map[int64]int64, error) {
var ip project_model.ProjectIssue var projIssues []project_model.ProjectIssue
has, err := db.GetEngine(ctx).Where("issue_id=?", issue.ID).Get(&ip) if err := db.GetEngine(ctx).Where("issue_id=?", issue.ID).Find(&projIssues); err != nil {
if err != nil { return nil, err
return 0, err
} else if !has {
return 0, nil
} }
return ip.ProjectColumnID, nil
result := make(map[int64]int64, len(projIssues))
for _, projIssue := range projIssues {
result[projIssue.ProjectID] = projIssue.ProjectColumnID
}
return result, nil
} }
func LoadProjectIssueColumnMap(ctx context.Context, projectID, defaultColumnID int64) (map[int64]int64, error) { func LoadProjectIssueColumnMap(ctx context.Context, projectID, defaultColumnID int64) (map[int64]int64, error) {
@@ -64,66 +61,91 @@ func LoadProjectIssueColumnMap(ctx context.Context, projectID, defaultColumnID i
return result, nil return result, nil
} }
// IssueAssignOrRemoveProject changes the project associated with an issue // IssueAssignOrRemoveProject updates the projects associated with an issue.
// If newProjectID is 0, the issue is removed from the project // It adds projects that are in newProjectIDs but not currently assigned,
func IssueAssignOrRemoveProject(ctx context.Context, issue *Issue, doer *user_model.User, newProjectID, newColumnID int64) error { // and removes projects that are currently assigned but not in newProjectIDs.
// If newProjectIDs is empty, all projects are removed from the issue.
// When adding an issue to a project, it is placed in the project's default column.
func IssueAssignOrRemoveProject(ctx context.Context, issue *Issue, doer *user_model.User, newProjectIDs []int64) error {
return db.WithTx(ctx, func(ctx context.Context) error { return db.WithTx(ctx, func(ctx context.Context) error {
oldProjectID := issue.projectID(ctx)
if err := issue.LoadRepo(ctx); err != nil { if err := issue.LoadRepo(ctx); err != nil {
return err return err
} }
// Only check if we add a new project and not remove it. oldProjectIDs, err := issue.projectIDs(ctx)
if newProjectID > 0 {
newProject, err := project_model.GetProjectByID(ctx, newProjectID)
if err != nil {
return err
}
if !newProject.CanBeAccessedByOwnerRepo(issue.Repo.OwnerID, issue.Repo) {
return util.NewPermissionDeniedErrorf("issue %d can't be accessed by project %d", issue.ID, newProject.ID)
}
if newColumnID == 0 {
newDefaultColumn, err := newProject.MustDefaultColumn(ctx)
if err != nil {
return err
}
newColumnID = newDefaultColumn.ID
}
}
if _, err := db.GetEngine(ctx).Where("project_issue.issue_id=?", issue.ID).Delete(&project_model.ProjectIssue{}); err != nil {
return err
}
if oldProjectID > 0 || newProjectID > 0 {
if _, err := CreateComment(ctx, &CreateCommentOptions{
Type: CommentTypeProject,
Doer: doer,
Repo: issue.Repo,
Issue: issue,
OldProjectID: oldProjectID,
ProjectID: newProjectID,
}); err != nil {
return err
}
}
if newProjectID == 0 {
return nil
}
if newColumnID == 0 {
panic("newColumnID must not be zero") // shouldn't happen
}
newSorting, err := project_model.GetColumnIssueNextSorting(ctx, newProjectID, newColumnID)
if err != nil { if err != nil {
return err return err
} }
return db.Insert(ctx, &project_model.ProjectIssue{
IssueID: issue.ID, projectsToAdd, projectsToRemove := util.DiffSlice(oldProjectIDs, newProjectIDs)
ProjectID: newProjectID, issue.isProjectsLoaded = false
ProjectColumnID: newColumnID, issue.Projects = nil
Sorting: newSorting,
}) if len(projectsToRemove) > 0 {
if _, err := db.GetEngine(ctx).Where("issue_id=?", issue.ID).In("project_id", projectsToRemove).Delete(&project_model.ProjectIssue{}); err != nil {
return err
}
for _, projectID := range projectsToRemove {
if _, err := CreateComment(ctx, &CreateCommentOptions{
Type: CommentTypeProject,
Doer: doer,
Repo: issue.Repo,
Issue: issue,
OldProjectID: projectID,
ProjectID: 0,
}); err != nil {
return err
}
}
}
if len(projectsToAdd) > 0 {
projectMap, err := project_model.GetProjectsMapByIDs(ctx, projectsToAdd)
if err != nil {
return err
}
for _, projectID := range projectsToAdd {
newProject, ok := projectMap[projectID]
if !ok {
return util.NewNotExistErrorf("project %d not found", projectID)
}
if !newProject.CanBeAccessedByOwnerRepo(issue.Repo.OwnerID, issue.Repo) {
return util.NewPermissionDeniedErrorf("issue %d can't be accessed by project %d", issue.ID, newProject.ID)
}
defaultColumn, err := newProject.MustDefaultColumn(ctx)
if err != nil {
return err
}
newSorting, err := project_model.GetColumnIssueNextSorting(ctx, projectID, defaultColumn.ID)
if err != nil {
return err
}
err = db.Insert(ctx, &project_model.ProjectIssue{
IssueID: issue.ID,
ProjectID: projectID,
ProjectColumnID: defaultColumn.ID,
Sorting: newSorting,
})
if err != nil {
return err
}
if _, err := CreateComment(ctx, &CreateCommentOptions{
Type: CommentTypeProject,
Doer: doer,
Repo: issue.Repo,
Issue: issue,
OldProjectID: 0,
ProjectID: projectID,
}); err != nil {
return err
}
}
}
return nil
}) })
} }

View File

@@ -0,0 +1,149 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package issues_test
import (
"fmt"
"testing"
issues_model "code.gitea.io/gitea/models/issues"
project_model "code.gitea.io/gitea/models/project"
"code.gitea.io/gitea/models/unittest"
user_model "code.gitea.io/gitea/models/user"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestIssueMultipleProjects(t *testing.T) {
require.NoError(t, unittest.PrepareTestDatabase())
t.Run("GeneralTest", func(t *testing.T) {
// Get test data
issue1 := unittest.AssertExistsAndLoadBean(t, &issues_model.Issue{ID: 1})
user2 := unittest.AssertExistsAndLoadBean(t, &user_model.User{ID: 2})
project1 := unittest.AssertExistsAndLoadBean(t, &project_model.Project{ID: 1})
// Create a second project for the same repository
project2 := &project_model.Project{
Title: "Test Project 2",
RepoID: issue1.RepoID,
Type: project_model.TypeRepository,
TemplateType: project_model.TemplateTypeBasicKanban,
}
require.NoError(t, project_model.NewProject(t.Context(), project2))
defer func() {
_ = project_model.DeleteProjectByID(t.Context(), project2.ID)
}()
err := issues_model.IssueAssignOrRemoveProject(t.Context(), issue1, user2, []int64{})
require.NoError(t, err)
err = issue1.LoadProjects(t.Context())
require.NoError(t, err)
require.Empty(t, issue1.Projects)
// assign issue to both projects (each project uses its own default column)
err = issues_model.IssueAssignOrRemoveProject(t.Context(), issue1, user2, []int64{project1.ID})
require.NoError(t, err)
assert.Nilf(t, issue1.Projects, "Issue's Projects should be nil after IssueAssignOrRemoveProject to ensure it reloads fresh data")
err = issue1.LoadProjects(t.Context())
require.NoError(t, err)
require.Len(t, issue1.Projects, 1)
err = issues_model.IssueAssignOrRemoveProject(t.Context(), issue1, user2, []int64{project1.ID, project2.ID})
require.NoError(t, err)
assert.Nilf(t, issue1.Projects, "Issue's Projects should be nil after IssueAssignOrRemoveProject to ensure it reloads fresh data")
err = issue1.LoadProjects(t.Context())
require.NoError(t, err)
require.Len(t, issue1.Projects, 2)
assert.ElementsMatch(t, []int64{project1.ID, project2.ID}, []int64{issue1.Projects[0].ID, issue1.Projects[1].ID}, "Issue should be in both projects")
// test issue's project column map
projectColumnMap, err := issue1.ProjectColumnMap(t.Context())
p1Col, _ := project1.MustDefaultColumn(t.Context())
p2Col, _ := project2.MustDefaultColumn(t.Context())
require.NoError(t, err)
assert.Equal(t, p1Col.ID, projectColumnMap[project1.ID])
assert.Equal(t, p2Col.ID, projectColumnMap[project2.ID])
// only keep project2
err = issues_model.IssueAssignOrRemoveProject(t.Context(), issue1, user2, []int64{project2.ID})
require.NoError(t, err)
err = issue1.LoadProjects(t.Context())
require.NoError(t, err)
require.Len(t, issue1.Projects, 1)
assert.Equal(t, project2.ID, issue1.Projects[0].ID)
// also test ResetAttributesLoaded
issue1.Projects = nil
issue1.ResetAttributesLoaded()
err = issue1.LoadProjects(t.Context())
require.NoError(t, err)
require.Len(t, issue1.Projects, 1)
assert.Equal(t, project2.ID, issue1.Projects[0].ID)
// remove issue's projects
err = issues_model.IssueAssignOrRemoveProject(t.Context(), issue1, user2, []int64{})
require.NoError(t, err)
err = issue1.LoadProjects(t.Context())
require.NoError(t, err)
require.Empty(t, issue1.Projects)
})
t.Run("QueryByMultipleProjectIDs", func(t *testing.T) {
// Get test data
issue1 := unittest.AssertExistsAndLoadBean(t, &issues_model.Issue{ID: 1})
issue2 := unittest.AssertExistsAndLoadBean(t, &issues_model.Issue{ID: 2})
user2 := unittest.AssertExistsAndLoadBean(t, &user_model.User{ID: 2})
// Create three projects
var projects []*project_model.Project
for i := 1; i <= 3; i++ {
project := &project_model.Project{
Title: fmt.Sprintf("Query Test Project %d", i),
RepoID: issue1.RepoID,
Type: project_model.TypeRepository,
TemplateType: project_model.TemplateTypeBasicKanban,
}
require.NoError(t, project_model.NewProject(t.Context(), project))
projects = append(projects, project)
defer func(id int64) {
_ = project_model.DeleteProjectByID(t.Context(), id)
}(project.ID)
}
// Assign issue1 to projects 1 and 2
err := issues_model.IssueAssignOrRemoveProject(t.Context(), issue1, user2, []int64{projects[0].ID, projects[1].ID})
require.NoError(t, err)
// Assign issue2 to project 3
err = issues_model.IssueAssignOrRemoveProject(t.Context(), issue2, user2, []int64{projects[2].ID})
require.NoError(t, err)
// Query for issues in project 3 only (should find issue2)
issues, err := issues_model.Issues(t.Context(), &issues_model.IssuesOptions{
RepoIDs: []int64{issue1.RepoID},
ProjectIDs: []int64{projects[2].ID},
})
require.NoError(t, err)
assert.NotEmpty(t, issues, "Should find issues in project 3")
// Verify issue2 is in the results
foundIssue2 := false
for _, issue := range issues {
if issue.ID == issue2.ID {
foundIssue2 = true
break
}
}
assert.True(t, foundIssue2, "Issue 2 should be found when querying project 3")
// FIXME: ISSUE-MULTIPLE-PROJECTS-FILTER: no multiple project filter support yet. Search logic is wrong. It should use "AND" but not "OR".
// Clean up
err = issues_model.IssueAssignOrRemoveProject(t.Context(), issue1, user2, []int64{})
require.NoError(t, err)
err = issues_model.IssueAssignOrRemoveProject(t.Context(), issue2, user2, []int64{})
require.NoError(t, err)
})
}

View File

@@ -16,6 +16,7 @@ import (
user_model "code.gitea.io/gitea/models/user" user_model "code.gitea.io/gitea/models/user"
"code.gitea.io/gitea/modules/container" "code.gitea.io/gitea/modules/container"
"code.gitea.io/gitea/modules/optional" "code.gitea.io/gitea/modules/optional"
"code.gitea.io/gitea/modules/util"
"xorm.io/builder" "xorm.io/builder"
"xorm.io/xorm" "xorm.io/xorm"
@@ -36,8 +37,7 @@ type IssuesOptions struct { //nolint:revive // export stutter
ReviewedID int64 ReviewedID int64
SubscriberID int64 SubscriberID int64
MilestoneIDs []int64 MilestoneIDs []int64
ProjectID int64 ProjectIDs []int64
ProjectColumnID int64
IsClosed optional.Option[bool] IsClosed optional.Option[bool]
IsPull optional.Option[bool] IsPull optional.Option[bool]
LabelIDs []int64 LabelIDs []int64
@@ -198,26 +198,19 @@ func applyMilestoneCondition(sess *xorm.Session, opts *IssuesOptions) {
} }
func applyProjectCondition(sess *xorm.Session, opts *IssuesOptions) { func applyProjectCondition(sess *xorm.Session, opts *IssuesOptions) {
if opts.ProjectID > 0 { // specific project projectIDs := util.SliceRemoveAll(opts.ProjectIDs, 0)
sess.Join("INNER", "project_issue", "issue.id = project_issue.issue_id"). if len(projectIDs) == 1 && projectIDs[0] == db.NoConditionID { // show those that are in no project
And("project_issue.project_id=?", opts.ProjectID) sess.And(builder.NotIn("issue.id", builder.Select("issue_id").From("project_issue")))
} else if opts.ProjectID == db.NoConditionID { // show those that are in no project } else if len(projectIDs) == 1 && projectIDs[0] > 0 { // single specific project
sess.And(builder.NotIn("issue.id", builder.Select("issue_id").From("project_issue").And(builder.Neq{"project_id": 0}))) sess.Join("INNER", "project_issue", "issue.id = project_issue.issue_id AND project_issue.project_id = ?", projectIDs[0])
} else if len(projectIDs) > 1 { // multiple projects
// FIXME: ISSUE-MULTIPLE-PROJECTS-FILTER: this logic is not right, it should use "AND" but not "OR"
sess.And(builder.In("issue.id", builder.Select("issue_id").From("project_issue").Where(builder.In("project_id", projectIDs))))
} }
// opts.ProjectID == 0 means all projects, // empty projectIDs means all projects,
// do not need to apply any condition // do not need to apply any condition
} }
func applyProjectColumnCondition(sess *xorm.Session, opts *IssuesOptions) {
// opts.ProjectColumnID == 0 means all project columns,
// do not need to apply any condition
if opts.ProjectColumnID > 0 {
sess.In("issue.id", builder.Select("issue_id").From("project_issue").Where(builder.Eq{"project_board_id": opts.ProjectColumnID}))
} else if opts.ProjectColumnID == db.NoConditionID {
sess.In("issue.id", builder.Select("issue_id").From("project_issue").Where(builder.Eq{"project_board_id": 0}))
}
}
func applyRepoConditions(sess *xorm.Session, opts *IssuesOptions) { func applyRepoConditions(sess *xorm.Session, opts *IssuesOptions) {
if len(opts.RepoIDs) == 1 { if len(opts.RepoIDs) == 1 {
opts.RepoCond = builder.Eq{"issue.repo_id": opts.RepoIDs[0]} opts.RepoCond = builder.Eq{"issue.repo_id": opts.RepoIDs[0]}
@@ -276,8 +269,6 @@ func applyConditions(sess *xorm.Session, opts *IssuesOptions) {
applyProjectCondition(sess, opts) applyProjectCondition(sess, opts)
applyProjectColumnCondition(sess, opts)
if opts.IsPull.Has() { if opts.IsPull.Has() {
sess.And("issue.is_pull=?", opts.IsPull.Value()) sess.And("issue.is_pull=?", opts.IsPull.Value())
} }

View File

@@ -424,10 +424,10 @@ func TestIssueLoadAttributes(t *testing.T) {
} }
if issue.ID == int64(1) { if issue.ID == int64(1) {
assert.Equal(t, int64(400), issue.TotalTrackedTime) assert.Equal(t, int64(400), issue.TotalTrackedTime)
assert.NotNil(t, issue.Project) assert.NotEmpty(t, issue.Projects)
assert.Equal(t, int64(1), issue.Project.ID) assert.Equal(t, int64(1), issue.Projects[0].ID)
} else { } else {
assert.Nil(t, issue.Project) assert.Empty(t, issue.Projects)
} }
} }
} }

View File

@@ -6,17 +6,18 @@ package base
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/migrationtest"
"code.gitea.io/gitea/modules/timeutil" "code.gitea.io/gitea/modules/timeutil"
"xorm.io/xorm/names" "xorm.io/xorm/names"
) )
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
MainTest(m) migrationtest.MainTest(m)
} }
func Test_DropTableColumns(t *testing.T) { func Test_DropTableColumns(t *testing.T) {
x, deferable := PrepareTestEnv(t, 0) x, deferable := migrationtest.PrepareTestEnv(t, 0)
defer deferable() defer deferable()
// FIXME: this logic seems wrong. Need to add an assertion here in the future, but it seems causing failure. // FIXME: this logic seems wrong. Need to add an assertion here in the future, but it seems causing failure.
if x == nil || t.Failed() { if x == nil || t.Failed() {

View File

@@ -1,223 +0,0 @@
// Copyright 2022 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package base
import (
"database/sql"
"fmt"
"os"
"path"
"path/filepath"
"testing"
"code.gitea.io/gitea/models/db"
"code.gitea.io/gitea/models/unittest"
"code.gitea.io/gitea/modules/git"
"code.gitea.io/gitea/modules/setting"
"code.gitea.io/gitea/modules/testlogger"
"code.gitea.io/gitea/modules/util"
"github.com/stretchr/testify/require"
"xorm.io/xorm"
"xorm.io/xorm/schemas"
)
// FIXME: this file shouldn't be in a normal package, it should only be compiled for tests
func newXORMEngine(t *testing.T) (*xorm.Engine, error) {
if err := db.InitEngine(t.Context()); err != nil {
return nil, err
}
x := unittest.GetXORMEngine()
return x, nil
}
func deleteDB() error {
switch {
case setting.Database.Type.IsSQLite3():
if err := util.Remove(setting.Database.Path); err != nil {
return err
}
return os.MkdirAll(path.Dir(setting.Database.Path), os.ModePerm)
case setting.Database.Type.IsMySQL():
db, err := sql.Open("mysql", fmt.Sprintf("%s:%s@tcp(%s)/",
setting.Database.User, setting.Database.Passwd, setting.Database.Host))
if err != nil {
return err
}
defer db.Close()
if _, err = db.Exec("DROP DATABASE IF EXISTS " + setting.Database.Name); err != nil {
return err
}
if _, err = db.Exec("CREATE DATABASE IF NOT EXISTS " + setting.Database.Name); err != nil {
return err
}
return nil
case setting.Database.Type.IsPostgreSQL():
db, err := sql.Open("postgres", fmt.Sprintf("postgres://%s:%s@%s/?sslmode=%s",
setting.Database.User, setting.Database.Passwd, setting.Database.Host, setting.Database.SSLMode))
if err != nil {
return err
}
defer db.Close()
if _, err = db.Exec("DROP DATABASE IF EXISTS " + setting.Database.Name); err != nil {
return err
}
if _, err = db.Exec("CREATE DATABASE " + setting.Database.Name); err != nil {
return err
}
db.Close()
// Check if we need to set up a specific schema
if len(setting.Database.Schema) != 0 {
db, err = sql.Open("postgres", fmt.Sprintf("postgres://%s:%s@%s/%s?sslmode=%s",
setting.Database.User, setting.Database.Passwd, setting.Database.Host, setting.Database.Name, setting.Database.SSLMode))
if err != nil {
return err
}
defer db.Close()
schrows, err := db.Query(fmt.Sprintf("SELECT 1 FROM information_schema.schemata WHERE schema_name = '%s'", setting.Database.Schema))
if err != nil {
return err
}
defer schrows.Close()
if !schrows.Next() {
// Create and set up a DB schema
_, err = db.Exec("CREATE SCHEMA " + setting.Database.Schema)
if err != nil {
return err
}
}
// Make the user's default search path the created schema; this will affect new connections
_, err = db.Exec(fmt.Sprintf(`ALTER USER "%s" SET search_path = %s`, setting.Database.User, setting.Database.Schema))
if err != nil {
return err
}
return nil
}
case setting.Database.Type.IsMSSQL():
host, port := setting.ParseMSSQLHostPort(setting.Database.Host)
db, err := sql.Open("mssql", fmt.Sprintf("server=%s; port=%s; database=%s; user id=%s; password=%s;",
host, port, "master", setting.Database.User, setting.Database.Passwd))
if err != nil {
return err
}
defer db.Close()
if _, err = db.Exec(fmt.Sprintf("DROP DATABASE IF EXISTS [%s]", setting.Database.Name)); err != nil {
return err
}
if _, err = db.Exec(fmt.Sprintf("CREATE DATABASE [%s]", setting.Database.Name)); err != nil {
return err
}
default:
return fmt.Errorf("unsupported database type: %s", setting.Database.Type)
}
return nil
}
// PrepareTestEnv prepares the test environment and reset the database. The skip parameter should usually be 0.
// Provide models to be sync'd with the database - in particular any models you expect fixtures to be loaded from.
//
// fixtures in `models/migrations/fixtures/<TestName>` will be loaded automatically
func PrepareTestEnv(t *testing.T, skip int, syncModels ...any) (*xorm.Engine, func()) {
t.Helper()
ourSkip := 2
ourSkip += skip
deferFn := testlogger.PrintCurrentTest(t, ourSkip)
giteaRoot := setting.GetGiteaTestSourceRoot()
require.NoError(t, unittest.SyncDirs(filepath.Join(giteaRoot, "tests/gitea-repositories-meta"), setting.RepoRootPath))
if err := deleteDB(); err != nil {
t.Fatalf("unable to reset database: %v", err)
return nil, deferFn
}
x, err := newXORMEngine(t)
require.NoError(t, err)
if x != nil {
oldDefer := deferFn
deferFn = func() {
oldDefer()
if err := x.Close(); err != nil {
t.Errorf("error during close: %v", err)
}
if err := deleteDB(); err != nil {
t.Errorf("unable to reset database: %v", err)
}
}
}
if err != nil {
return x, deferFn
}
if len(syncModels) > 0 {
if err := x.Sync(syncModels...); err != nil {
t.Errorf("error during sync: %v", err)
return x, deferFn
}
}
fixturesDir := filepath.Join(giteaRoot, "models", "migrations", "fixtures", t.Name())
if _, err := os.Stat(fixturesDir); err == nil {
t.Logf("initializing fixtures from: %s", fixturesDir)
if err := unittest.InitFixtures(
unittest.FixturesOptions{
Dir: fixturesDir,
}, x); err != nil {
t.Errorf("error whilst initializing fixtures from %s: %v", fixturesDir, err)
return x, deferFn
}
if err := unittest.LoadFixtures(); err != nil {
t.Errorf("error whilst loading fixtures from %s: %v", fixturesDir, err)
return x, deferFn
}
} else if !os.IsNotExist(err) {
t.Errorf("unexpected error whilst checking for existence of fixtures: %v", err)
} else {
t.Logf("no fixtures found in: %s", fixturesDir)
}
return x, deferFn
}
func LoadTableSchemasMap(t *testing.T, x *xorm.Engine) map[string]*schemas.Table {
tables, err := x.DBMetas()
require.NoError(t, err)
tableMap := make(map[string]*schemas.Table)
for _, table := range tables {
tableMap[table.Name] = table
}
return tableMap
}
func mainTest(m *testing.M) int {
testlogger.Init()
err := setting.PrepareIntegrationTestConfig()
if err != nil {
return testlogger.MainErrorf("Unable to prepare integration test config: %v", err)
}
setting.SetupGiteaTestEnv()
if err = git.InitFull(); err != nil {
return testlogger.MainErrorf("Unable to InitFull: %v", err)
}
setting.LoadDBSetting()
setting.InitLoggersForTest()
return m.Run()
}
func MainTest(m *testing.M) {
os.Exit(mainTest(m))
}

View File

@@ -0,0 +1,120 @@
// Copyright 2022 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package migrationtest
import (
"os"
"path/filepath"
"testing"
"code.gitea.io/gitea/models/db"
"code.gitea.io/gitea/models/unittest"
"code.gitea.io/gitea/modules/git"
"code.gitea.io/gitea/modules/setting"
"code.gitea.io/gitea/modules/testlogger"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"xorm.io/xorm"
"xorm.io/xorm/schemas"
)
// PrepareTestEnv prepares the test environment and reset the database. The skip parameter should usually be 0.
// Provide models to be sync'd with the database - in particular any models you expect fixtures to be loaded from.
//
// fixtures in `models/migrations/fixtures/<TestName>` will be loaded automatically
func PrepareTestEnv(t *testing.T, skip int, syncModels ...any) (*xorm.Engine, func()) {
t.Helper()
ourSkip := 2
ourSkip += skip
deferFn := testlogger.PrintCurrentTest(t, ourSkip)
giteaRoot := setting.GetGiteaTestSourceRoot()
require.NoError(t, unittest.SyncDirs(filepath.Join(giteaRoot, "tests/gitea-repositories-meta"), setting.RepoRootPath))
cleanup, err := unittest.ResetTestDatabase()
if err != nil {
t.Fatalf("unable to reset database: %v", err)
return nil, deferFn
}
{
oldDefer := deferFn
deferFn = func() {
cleanup()
oldDefer()
}
}
err = db.InitEngine(t.Context())
if !assert.NoError(t, err) {
return nil, deferFn
}
x := unittest.GetXORMEngine()
{
oldDefer := deferFn
deferFn = func() {
_ = x.Close()
oldDefer()
}
}
if len(syncModels) > 0 {
if err := x.Sync(syncModels...); err != nil {
t.Errorf("error during sync: %v", err)
return x, deferFn
}
}
fixturesDir := filepath.Join(giteaRoot, "models", "migrations", "fixtures", t.Name())
if _, err := os.Stat(fixturesDir); err == nil {
t.Logf("initializing fixtures from: %s", fixturesDir)
if err := unittest.InitFixtures(
unittest.FixturesOptions{
Dir: fixturesDir,
}, x); err != nil {
t.Errorf("error whilst initializing fixtures from %s: %v", fixturesDir, err)
return x, deferFn
}
if err := unittest.LoadFixtures(); err != nil {
t.Errorf("error whilst loading fixtures from %s: %v", fixturesDir, err)
return x, deferFn
}
} else if !os.IsNotExist(err) {
t.Errorf("unexpected error whilst checking for existence of fixtures: %v", err)
} else {
t.Logf("no fixtures found in: %s", fixturesDir)
}
return x, deferFn
}
func LoadTableSchemasMap(t *testing.T, x *xorm.Engine) map[string]*schemas.Table {
tables, err := x.DBMetas()
require.NoError(t, err)
tableMap := make(map[string]*schemas.Table)
for _, table := range tables {
tableMap[table.Name] = table
}
return tableMap
}
func mainTest(m *testing.M) int {
testlogger.Init()
err := setting.PrepareIntegrationTestConfig()
if err != nil {
return testlogger.MainErrorf("Unable to prepare integration test config: %v", err)
}
setting.SetupGiteaTestEnv()
if err = git.InitFull(); err != nil {
return testlogger.MainErrorf("Unable to InitFull: %v", err)
}
setting.LoadDBSetting()
setting.InitLoggersForTest()
return m.Run()
}
func MainTest(m *testing.M) {
os.Exit(mainTest(m))
}

View File

@@ -6,9 +6,9 @@ package v1_14
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
) )
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
base.MainTest(m) migrationtest.MainTest(m)
} }

View File

@@ -6,7 +6,7 @@ package v1_14
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
) )
@@ -47,7 +47,7 @@ func Test_RemoveInvalidLabels(t *testing.T) {
} }
// load and prepare the test database // load and prepare the test database
x, deferable := base.PrepareTestEnv(t, 0, new(Comment), new(Issue), new(Repository), new(IssueLabel), new(Label)) x, deferable := migrationtest.PrepareTestEnv(t, 0, new(Comment), new(Issue), new(Repository), new(IssueLabel), new(Label))
if x == nil || t.Failed() { if x == nil || t.Failed() {
defer deferable() defer deferable()
return return

View File

@@ -6,7 +6,7 @@ package v1_14
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"code.gitea.io/gitea/modules/timeutil" "code.gitea.io/gitea/modules/timeutil"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
@@ -34,7 +34,7 @@ func Test_DeleteOrphanedIssueLabels(t *testing.T) {
} }
// Prepare and load the testing database // Prepare and load the testing database
x, deferable := base.PrepareTestEnv(t, 0, new(IssueLabel), new(Label)) x, deferable := migrationtest.PrepareTestEnv(t, 0, new(IssueLabel), new(Label))
if x == nil || t.Failed() { if x == nil || t.Failed() {
defer deferable() defer deferable()
return return

View File

@@ -6,9 +6,9 @@ package v1_15
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
) )
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
base.MainTest(m) migrationtest.MainTest(m)
} }

View File

@@ -7,7 +7,7 @@ import (
"strings" "strings"
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
) )
@@ -20,7 +20,7 @@ func Test_AddPrimaryEmail2EmailAddress(t *testing.T) {
} }
// Prepare and load the testing database // Prepare and load the testing database
x, deferable := base.PrepareTestEnv(t, 0, new(User)) x, deferable := migrationtest.PrepareTestEnv(t, 0, new(User))
if x == nil || t.Failed() { if x == nil || t.Failed() {
defer deferable() defer deferable()
return return

View File

@@ -6,7 +6,7 @@ package v1_15
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
) )
@@ -20,7 +20,7 @@ func Test_AddIssueResourceIndexTable(t *testing.T) {
} }
// Prepare and load the testing database // Prepare and load the testing database
x, deferable := base.PrepareTestEnv(t, 0, new(Issue)) x, deferable := migrationtest.PrepareTestEnv(t, 0, new(Issue))
if x == nil || t.Failed() { if x == nil || t.Failed() {
defer deferable() defer deferable()
return return

View File

@@ -6,9 +6,9 @@ package v1_16
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
) )
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
base.MainTest(m) migrationtest.MainTest(m)
} }

View File

@@ -6,7 +6,7 @@ package v1_16
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"code.gitea.io/gitea/modules/json" "code.gitea.io/gitea/modules/json"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
@@ -27,7 +27,7 @@ func (ls *LoginSourceOriginalV189) TableName() string {
func Test_UnwrapLDAPSourceCfg(t *testing.T) { func Test_UnwrapLDAPSourceCfg(t *testing.T) {
// Prepare and load the testing database // Prepare and load the testing database
x, deferable := base.PrepareTestEnv(t, 0, new(LoginSourceOriginalV189)) x, deferable := migrationtest.PrepareTestEnv(t, 0, new(LoginSourceOriginalV189))
if x == nil || t.Failed() { if x == nil || t.Failed() {
defer deferable() defer deferable()
return return

View File

@@ -6,7 +6,7 @@ package v1_16
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
) )
@@ -31,7 +31,7 @@ func Test_AddRepoIDForAttachment(t *testing.T) {
} }
// Prepare and load the testing database // Prepare and load the testing database
x, deferrable := base.PrepareTestEnv(t, 0, new(Attachment), new(Issue), new(Release)) x, deferrable := migrationtest.PrepareTestEnv(t, 0, new(Attachment), new(Issue), new(Release))
defer deferrable() defer deferrable()
if x == nil || t.Failed() { if x == nil || t.Failed() {
return return

View File

@@ -6,7 +6,7 @@ package v1_16
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
) )
@@ -21,7 +21,7 @@ func Test_AddTableCommitStatusIndex(t *testing.T) {
} }
// Prepare and load the testing database // Prepare and load the testing database
x, deferable := base.PrepareTestEnv(t, 0, new(CommitStatus)) x, deferable := migrationtest.PrepareTestEnv(t, 0, new(CommitStatus))
if x == nil || t.Failed() { if x == nil || t.Failed() {
defer deferable() defer deferable()
return return

View File

@@ -6,7 +6,7 @@ package v1_16
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"code.gitea.io/gitea/modules/timeutil" "code.gitea.io/gitea/modules/timeutil"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
@@ -44,7 +44,7 @@ func Test_RemigrateU2FCredentials(t *testing.T) {
} }
// Prepare and load the testing database // Prepare and load the testing database
x, deferable := base.PrepareTestEnv(t, 0, new(WebauthnCredential), new(U2fRegistration), new(ExpectedWebauthnCredential)) x, deferable := migrationtest.PrepareTestEnv(t, 0, new(WebauthnCredential), new(U2fRegistration), new(ExpectedWebauthnCredential))
if x == nil || t.Failed() { if x == nil || t.Failed() {
defer deferable() defer deferable()
return return

View File

@@ -6,9 +6,9 @@ package v1_17
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
) )
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
base.MainTest(m) migrationtest.MainTest(m)
} }

View File

@@ -7,7 +7,7 @@ import (
"encoding/base32" "encoding/base32"
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
) )
@@ -38,7 +38,7 @@ func Test_StoreWebauthnCredentialIDAsBytes(t *testing.T) {
} }
// Prepare and load the testing database // Prepare and load the testing database
x, deferable := base.PrepareTestEnv(t, 0, new(WebauthnCredential), new(ExpectedWebauthnCredential)) x, deferable := migrationtest.PrepareTestEnv(t, 0, new(WebauthnCredential), new(ExpectedWebauthnCredential))
defer deferable() defer deferable()
if x == nil || t.Failed() { if x == nil || t.Failed() {
return return

View File

@@ -6,9 +6,9 @@ package v1_18
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
) )
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
base.MainTest(m) migrationtest.MainTest(m)
} }

View File

@@ -7,7 +7,7 @@ import (
"testing" "testing"
"code.gitea.io/gitea/models/issues" "code.gitea.io/gitea/models/issues"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
) )
@@ -16,7 +16,7 @@ func Test_UpdateOpenMilestoneCounts(t *testing.T) {
type ExpectedMilestone issues.Milestone type ExpectedMilestone issues.Milestone
// Prepare and load the testing database // Prepare and load the testing database
x, deferable := base.PrepareTestEnv(t, 0, new(issues.Milestone), new(ExpectedMilestone), new(issues.Issue)) x, deferable := migrationtest.PrepareTestEnv(t, 0, new(issues.Milestone), new(ExpectedMilestone), new(issues.Issue))
defer deferable() defer deferable()
if x == nil || t.Failed() { if x == nil || t.Failed() {
return return

View File

@@ -6,7 +6,7 @@ package v1_18
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
) )
@@ -18,7 +18,7 @@ func Test_AddConfidentialClientColumnToOAuth2ApplicationTable(t *testing.T) {
} }
// Prepare and load the testing database // Prepare and load the testing database
x, deferable := base.PrepareTestEnv(t, 0, new(oauth2Application)) x, deferable := migrationtest.PrepareTestEnv(t, 0, new(oauth2Application))
defer deferable() defer deferable()
if x == nil || t.Failed() { if x == nil || t.Failed() {
return return

View File

@@ -6,9 +6,9 @@ package v1_19
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
) )
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
base.MainTest(m) migrationtest.MainTest(m)
} }

View File

@@ -6,7 +6,7 @@ package v1_19
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"code.gitea.io/gitea/modules/json" "code.gitea.io/gitea/modules/json"
"code.gitea.io/gitea/modules/secret" "code.gitea.io/gitea/modules/secret"
"code.gitea.io/gitea/modules/setting" "code.gitea.io/gitea/modules/setting"
@@ -39,7 +39,7 @@ func Test_AddHeaderAuthorizationEncryptedColWebhook(t *testing.T) {
} }
// Prepare and load the testing database // Prepare and load the testing database
x, deferable := base.PrepareTestEnv(t, 0, new(Webhook), new(ExpectedWebhook), new(HookTask)) x, deferable := migrationtest.PrepareTestEnv(t, 0, new(Webhook), new(ExpectedWebhook), new(HookTask))
defer deferable() defer deferable()
if x == nil || t.Failed() { if x == nil || t.Failed() {
return return

View File

@@ -6,9 +6,9 @@ package v1_20
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
) )
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
base.MainTest(m) migrationtest.MainTest(m)
} }

View File

@@ -8,7 +8,7 @@ import (
"strings" "strings"
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
) )
@@ -66,7 +66,7 @@ func Test_ConvertScopedAccessTokens(t *testing.T) {
}) })
} }
x, deferable := base.PrepareTestEnv(t, 0, new(AccessToken)) x, deferable := migrationtest.PrepareTestEnv(t, 0, new(AccessToken))
defer deferable() defer deferable()
if x == nil || t.Failed() { if x == nil || t.Failed() {
t.Skip() t.Skip()

View File

@@ -6,9 +6,9 @@ package v1_21
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
) )
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
base.MainTest(m) migrationtest.MainTest(m)
} }

View File

@@ -6,9 +6,9 @@ package v1_22
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
) )
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
base.MainTest(m) migrationtest.MainTest(m)
} }

View File

@@ -6,7 +6,7 @@ package v1_22
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
) )
@@ -21,7 +21,7 @@ func Test_AddCombinedIndexToIssueUser(t *testing.T) {
} }
// Prepare and load the testing database // Prepare and load the testing database
x, deferable := base.PrepareTestEnv(t, 0, new(IssueUser)) x, deferable := migrationtest.PrepareTestEnv(t, 0, new(IssueUser))
defer deferable() defer deferable()
assert.NoError(t, AddCombinedIndexToIssueUser(x)) assert.NoError(t, AddCombinedIndexToIssueUser(x))

View File

@@ -6,7 +6,7 @@ package v1_22
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"xorm.io/xorm" "xorm.io/xorm"
@@ -64,7 +64,7 @@ func PrepareOldRepository(t *testing.T) (*xorm.Engine, func()) {
} }
// Prepare and load the testing database // Prepare and load the testing database
return base.PrepareTestEnv(t, 0, return migrationtest.PrepareTestEnv(t, 0,
new(Repository), new(Repository),
new(CommitStatus), new(CommitStatus),
new(RepoArchiver), new(RepoArchiver),

View File

@@ -7,7 +7,7 @@ import (
"strconv" "strconv"
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
) )
@@ -20,7 +20,7 @@ func Test_UpdateBadgeColName(t *testing.T) {
} }
// Prepare and load the testing database // Prepare and load the testing database
x, deferable := base.PrepareTestEnv(t, 0, new(Badge)) x, deferable := migrationtest.PrepareTestEnv(t, 0, new(Badge))
defer deferable() defer deferable()
if x == nil || t.Failed() { if x == nil || t.Failed() {
return return

View File

@@ -6,7 +6,7 @@ package v1_22
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"code.gitea.io/gitea/models/project" "code.gitea.io/gitea/models/project"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
@@ -14,7 +14,7 @@ import (
func Test_CheckProjectColumnsConsistency(t *testing.T) { func Test_CheckProjectColumnsConsistency(t *testing.T) {
// Prepare and load the testing database // Prepare and load the testing database
x, deferable := base.PrepareTestEnv(t, 0, new(project.Project), new(project.Column)) x, deferable := migrationtest.PrepareTestEnv(t, 0, new(project.Project), new(project.Column))
defer deferable() defer deferable()
if x == nil || t.Failed() { if x == nil || t.Failed() {
return return

View File

@@ -6,7 +6,7 @@ package v1_22
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"xorm.io/xorm/schemas" "xorm.io/xorm/schemas"
@@ -20,7 +20,7 @@ func Test_AddUniqueIndexForProjectIssue(t *testing.T) {
} }
// Prepare and load the testing database // Prepare and load the testing database
x, deferable := base.PrepareTestEnv(t, 0, new(ProjectIssue)) x, deferable := migrationtest.PrepareTestEnv(t, 0, new(ProjectIssue))
defer deferable() defer deferable()
if x == nil || t.Failed() { if x == nil || t.Failed() {
return return

View File

@@ -6,9 +6,9 @@ package v1_23
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
) )
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
base.MainTest(m) migrationtest.MainTest(m)
} }

View File

@@ -6,7 +6,7 @@ package v1_23
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"code.gitea.io/gitea/modules/timeutil" "code.gitea.io/gitea/modules/timeutil"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
@@ -44,7 +44,7 @@ func Test_AddIndexToActionTaskStoppedLogExpired(t *testing.T) {
} }
// Prepare and load the testing database // Prepare and load the testing database
x, deferable := base.PrepareTestEnv(t, 0, new(ActionTask)) x, deferable := migrationtest.PrepareTestEnv(t, 0, new(ActionTask))
defer deferable() defer deferable()
assert.NoError(t, AddIndexToActionTaskStoppedLogExpired(x)) assert.NoError(t, AddIndexToActionTaskStoppedLogExpired(x))

View File

@@ -6,7 +6,7 @@ package v1_23
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"code.gitea.io/gitea/modules/timeutil" "code.gitea.io/gitea/modules/timeutil"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
@@ -33,7 +33,7 @@ func Test_AddIndexForReleaseSha1(t *testing.T) {
} }
// Prepare and load the testing database // Prepare and load the testing database
x, deferable := base.PrepareTestEnv(t, 0, new(Release)) x, deferable := migrationtest.PrepareTestEnv(t, 0, new(Release))
defer deferable() defer deferable()
assert.NoError(t, AddIndexForReleaseSha1(x)) assert.NoError(t, AddIndexForReleaseSha1(x))

View File

@@ -6,9 +6,9 @@ package v1_25
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
) )
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
base.MainTest(m) migrationtest.MainTest(m)
} }

View File

@@ -6,7 +6,7 @@ package v1_25
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"code.gitea.io/gitea/modules/setting" "code.gitea.io/gitea/modules/setting"
"code.gitea.io/gitea/modules/timeutil" "code.gitea.io/gitea/modules/timeutil"
@@ -44,12 +44,12 @@ func Test_UseLongTextInSomeColumnsAndFixBugs(t *testing.T) {
} }
// Prepare and load the testing database // Prepare and load the testing database
x, deferrable := base.PrepareTestEnv(t, 0, new(ReviewState), new(PackageProperty), new(Notice)) x, deferrable := migrationtest.PrepareTestEnv(t, 0, new(ReviewState), new(PackageProperty), new(Notice))
defer deferrable() defer deferrable()
require.NoError(t, UseLongTextInSomeColumnsAndFixBugs(x)) require.NoError(t, UseLongTextInSomeColumnsAndFixBugs(x))
tables := base.LoadTableSchemasMap(t, x) tables := migrationtest.LoadTableSchemasMap(t, x)
table := tables["review_state"] table := tables["review_state"]
column := table.GetColumn("updated_files") column := table.GetColumn("updated_files")
assert.Equal(t, "LONGTEXT", column.SQLType.Name) assert.Equal(t, "LONGTEXT", column.SQLType.Name)

View File

@@ -6,7 +6,7 @@ package v1_25
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"code.gitea.io/gitea/modules/setting" "code.gitea.io/gitea/modules/setting"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
@@ -23,11 +23,11 @@ func Test_ExtendCommentTreePathLength(t *testing.T) {
TreePath string `xorm:"VARCHAR(255)"` TreePath string `xorm:"VARCHAR(255)"`
} }
x, deferrable := base.PrepareTestEnv(t, 0, new(Comment)) x, deferrable := migrationtest.PrepareTestEnv(t, 0, new(Comment))
defer deferrable() defer deferrable()
require.NoError(t, ExtendCommentTreePathLength(x)) require.NoError(t, ExtendCommentTreePathLength(x))
table := base.LoadTableSchemasMap(t, x)["comment"] table := migrationtest.LoadTableSchemasMap(t, x)["comment"]
column := table.GetColumn("tree_path") column := table.GetColumn("tree_path")
assert.Contains(t, []string{"NVARCHAR", "VARCHAR"}, column.SQLType.Name) assert.Contains(t, []string{"NVARCHAR", "VARCHAR"}, column.SQLType.Name)
assert.EqualValues(t, 4000, column.Length) assert.EqualValues(t, 4000, column.Length)

View File

@@ -6,9 +6,9 @@ package v1_26
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
) )
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
base.MainTest(m) migrationtest.MainTest(m)
} }

View File

@@ -6,7 +6,7 @@ package v1_26
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"code.gitea.io/gitea/modules/timeutil" "code.gitea.io/gitea/modules/timeutil"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
@@ -38,7 +38,7 @@ func Test_FixMissedRepoIDWhenMigrateAttachments(t *testing.T) {
} }
// Prepare and load the testing database // Prepare and load the testing database
x, deferrable := base.PrepareTestEnv(t, 0, new(Attachment), new(Issue), new(Release)) x, deferrable := migrationtest.PrepareTestEnv(t, 0, new(Attachment), new(Issue), new(Release))
defer deferrable() defer deferrable()
require.NoError(t, FixMissedRepoIDWhenMigrateAttachments(x)) require.NoError(t, FixMissedRepoIDWhenMigrateAttachments(x))

View File

@@ -6,7 +6,7 @@ package v1_26
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"code.gitea.io/gitea/modules/setting" "code.gitea.io/gitea/modules/setting"
"code.gitea.io/gitea/modules/test" "code.gitea.io/gitea/modules/test"
@@ -57,7 +57,7 @@ func Test_FixCommitStatusTargetURLToUseRunAndJobID(t *testing.T) {
TargetURL string TargetURL string
} }
x, deferable := base.PrepareTestEnv(t, 0, x, deferable := migrationtest.PrepareTestEnv(t, 0,
new(Repository), new(Repository),
new(ActionRun), new(ActionRun),
new(ActionRunJob), new(ActionRunJob),

View File

@@ -6,7 +6,7 @@ package v1_26
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
@@ -17,7 +17,7 @@ func Test_AddDisabledToActionRunner(t *testing.T) {
Name string Name string
} }
x, deferable := base.PrepareTestEnv(t, 0, new(ActionRunner)) x, deferable := migrationtest.PrepareTestEnv(t, 0, new(ActionRunner))
defer deferable() defer deferable()
_, err := x.Insert(&ActionRunner{Name: "runner"}) _, err := x.Insert(&ActionRunner{Name: "runner"})

View File

@@ -6,7 +6,7 @@ package v1_26
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
) )
@@ -22,7 +22,7 @@ func (UserBadgeBefore) TableName() string {
} }
func Test_AddUniqueIndexForUserBadge(t *testing.T) { func Test_AddUniqueIndexForUserBadge(t *testing.T) {
x, deferable := base.PrepareTestEnv(t, 0, new(UserBadgeBefore)) x, deferable := migrationtest.PrepareTestEnv(t, 0, new(UserBadgeBefore))
defer deferable() defer deferable()
if x == nil || t.Failed() { if x == nil || t.Failed() {
return return

View File

@@ -6,9 +6,9 @@ package v1_27
import ( import (
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
) )
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
base.MainTest(m) migrationtest.MainTest(m)
} }

View File

@@ -8,7 +8,7 @@ import (
"slices" "slices"
"testing" "testing"
"code.gitea.io/gitea/models/migrations/base" "code.gitea.io/gitea/models/migrations/migrationtest"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
@@ -49,7 +49,7 @@ func (actionArtifactBeforeV331) TableName() string {
} }
func Test_AddActionRunAttemptModel(t *testing.T) { func Test_AddActionRunAttemptModel(t *testing.T) {
x, deferable := base.PrepareTestEnv(t, 0, x, deferable := migrationtest.PrepareTestEnv(t, 0,
new(actionRunBeforeV331), new(actionRunBeforeV331),
new(actionRunJobBeforeV331), new(actionRunJobBeforeV331),
new(actionArtifactBeforeV331), new(actionArtifactBeforeV331),
@@ -69,7 +69,7 @@ func Test_AddActionRunAttemptModel(t *testing.T) {
require.NoError(t, AddActionRunAttemptModel(x)) require.NoError(t, AddActionRunAttemptModel(x))
tableMap := base.LoadTableSchemasMap(t, x) tableMap := migrationtest.LoadTableSchemasMap(t, x)
attemptTable := tableMap["action_run_attempt"] attemptTable := tableMap["action_run_attempt"]
require.NotNil(t, attemptTable) require.NotNil(t, attemptTable)

View File

@@ -302,6 +302,15 @@ func GetProjectByID(ctx context.Context, id int64) (*Project, error) {
return p, nil return p, nil
} }
// GetProjectsMapByIDs returns projects by a list of IDs.
func GetProjectsMapByIDs(ctx context.Context, ids []int64) (map[int64]*Project, error) {
projects := make(map[int64]*Project, len(ids))
if len(ids) == 0 {
return projects, nil
}
return projects, db.GetEngine(ctx).In("id", ids).Find(&projects)
}
func GetProjectByIDAndOwner(ctx context.Context, id, ownerID int64) (*Project, error) { func GetProjectByIDAndOwner(ctx context.Context, id, ownerID int64) (*Project, error) {
p := new(Project) p := new(Project)
has, err := db.GetEngine(ctx).ID(id).And("owner_id = ?", ownerID).Get(p) has, err := db.GetEngine(ctx).ID(id).And("owner_id = ?", ownerID).Get(p)

View File

@@ -5,6 +5,8 @@ package unittest
import ( import (
"context" "context"
"database/sql"
"errors"
"fmt" "fmt"
"os" "os"
"path/filepath" "path/filepath"
@@ -102,6 +104,101 @@ func mainTest(m *testing.M, testOptsArg ...*TestOptions) int {
return exitStatus return exitStatus
} }
func ResetTestDatabase() (cleanup func(), err error) {
defer func() {
if cleanup == nil {
cleanup = func() {}
}
}()
connOpts := db.GlobalConnOptions()
driverDefault, connStrDefault, err := db.ConnStrDefaultDatabase(connOpts)
if err != nil {
return nil, err
}
driverDatabase, connStrDatabase, err := db.ConnStr(connOpts)
if err != nil {
return nil, err
}
if connOpts.Type.IsSQLite3() {
if !strings.HasSuffix(connOpts.SQLitePath, "-test.db") {
return nil, errors.New(`testing database file for sqlite3 must end in "-test.db"`)
}
_ = os.Remove(connOpts.SQLitePath)
err = os.MkdirAll(filepath.Dir(connOpts.SQLitePath), os.ModePerm)
if err != nil {
return nil, err
}
cleanup = func() {
_ = os.Remove(connOpts.SQLitePath)
_ = os.Remove(filepath.Dir(connOpts.SQLitePath))
}
return cleanup, nil
}
if !strings.Contains(connOpts.Database, "test") {
return nil, fmt.Errorf(`testing database name for %s must contain "test"`, connOpts.Database)
}
quotedDbName := connOpts.Database
if connOpts.Type.IsMSSQL() {
quotedDbName = `[` + connOpts.Database + `]`
}
sqlExec := func(sqlDB *sql.DB, sql string) error {
_, err := sqlDB.Exec(sql)
if err != nil {
return fmt.Errorf("failed to execute SQL %q: %w", sql, err)
}
return nil
}
createDatabase := func() error {
sqlDB, err := sql.Open(driverDefault, connStrDefault)
if err != nil {
return err
}
defer sqlDB.Close()
if err = sqlExec(sqlDB, "DROP DATABASE IF EXISTS "+quotedDbName); err != nil {
return err
}
return sqlExec(sqlDB, "CREATE DATABASE "+quotedDbName)
}
if err = createDatabase(); err != nil {
return nil, err
}
cleanup = func() {
sqlDB, err := sql.Open(driverDefault, connStrDefault)
if err != nil {
return
}
defer sqlDB.Close()
_, _ = sqlDB.Exec("DROP DATABASE IF EXISTS " + quotedDbName)
}
createDatabaseSchema := func() error {
if !connOpts.Type.IsPostgreSQL() {
return nil
}
if connOpts.Schema == "" {
return nil
}
sqlDB, err := sql.Open(driverDatabase, connStrDatabase)
if err != nil {
return err
}
defer sqlDB.Close()
if err = sqlExec(sqlDB, "DROP SCHEMA IF EXISTS "+connOpts.Schema); err != nil {
return err
}
return sqlExec(sqlDB, "CREATE SCHEMA "+connOpts.Schema)
}
return cleanup, createDatabaseSchema()
}
// FixturesOptions fixtures needs to be loaded options // FixturesOptions fixtures needs to be loaded options
type FixturesOptions struct { type FixturesOptions struct {
Dir string Dir string
@@ -110,11 +207,12 @@ type FixturesOptions struct {
// CreateTestEngine creates a memory database and loads the fixture data from fixturesDir // CreateTestEngine creates a memory database and loads the fixture data from fixturesDir
func CreateTestEngine(opts FixturesOptions) error { func CreateTestEngine(opts FixturesOptions) error {
x, err := xorm.NewEngine("sqlite3", "file::memory:?cache=shared&_txlock=immediate") driver, connStr, err := db.ConnStr(db.ConnOptions{Type: "sqlite3", SQLitePath: ":memory:"})
if err != nil {
return err
}
x, err := xorm.NewEngine(driver, connStr)
if err != nil { if err != nil {
if strings.Contains(err.Error(), "unknown driver") {
return fmt.Errorf("sqlite3 requires: -tags sqlite,sqlite_unlock_notify\n%w", err)
}
return err return err
} }
x.SetMapper(names.GonicMapper{}) x.SetMapper(names.GonicMapper{})

View File

@@ -18,8 +18,7 @@ import (
"code.gitea.io/gitea/modules/gitrepo" "code.gitea.io/gitea/modules/gitrepo"
"code.gitea.io/gitea/modules/indexer" "code.gitea.io/gitea/modules/indexer"
"code.gitea.io/gitea/modules/indexer/code/internal" "code.gitea.io/gitea/modules/indexer/code/internal"
indexer_internal "code.gitea.io/gitea/modules/indexer/internal" es "code.gitea.io/gitea/modules/indexer/internal/elasticsearch"
inner_elasticsearch "code.gitea.io/gitea/modules/indexer/internal/elasticsearch"
"code.gitea.io/gitea/modules/json" "code.gitea.io/gitea/modules/json"
"code.gitea.io/gitea/modules/log" "code.gitea.io/gitea/modules/log"
"code.gitea.io/gitea/modules/setting" "code.gitea.io/gitea/modules/setting"
@@ -28,23 +27,15 @@ import (
"code.gitea.io/gitea/modules/util" "code.gitea.io/gitea/modules/util"
"github.com/go-enry/go-enry/v2" "github.com/go-enry/go-enry/v2"
"github.com/olivere/elastic/v7"
) )
const ( const esRepoIndexerLatestVersion = 3
esRepoIndexerLatestVersion = 3
// multi-match-types, currently only 2 types are used
// Reference: https://www.elastic.co/guide/en/elasticsearch/reference/7.0/query-dsl-multi-match-query.html#multi-match-types
esMultiMatchTypeBestFields = "best_fields"
esMultiMatchTypePhrasePrefix = "phrase_prefix"
)
var _ internal.Indexer = &Indexer{} var _ internal.Indexer = &Indexer{}
// Indexer implements Indexer interface // Indexer implements Indexer interface
type Indexer struct { type Indexer struct {
inner *inner_elasticsearch.Indexer *es.Indexer
indexer_internal.Indexer // do not composite inner_elasticsearch.Indexer directly to avoid exposing too much
} }
func (b *Indexer) SupportedSearchModes() []indexer.SearchMode { func (b *Indexer) SupportedSearchModes() []indexer.SearchMode {
@@ -53,12 +44,7 @@ func (b *Indexer) SupportedSearchModes() []indexer.SearchMode {
// NewIndexer creates a new elasticsearch indexer // NewIndexer creates a new elasticsearch indexer
func NewIndexer(url, indexerName string) *Indexer { func NewIndexer(url, indexerName string) *Indexer {
inner := inner_elasticsearch.NewIndexer(url, indexerName, esRepoIndexerLatestVersion, defaultMapping) return &Indexer{Indexer: es.NewIndexer(url, indexerName, esRepoIndexerLatestVersion, defaultMapping)}
indexer := &Indexer{
inner: inner,
Indexer: inner,
}
return indexer
} }
const ( const (
@@ -138,7 +124,7 @@ const (
}` }`
) )
func (b *Indexer) addUpdate(ctx context.Context, catFileBatch git.CatFileBatch, sha string, update internal.FileUpdate, repo *repo_model.Repository) ([]elastic.BulkableRequest, error) { func (b *Indexer) addUpdate(ctx context.Context, catFileBatch git.CatFileBatch, sha string, update internal.FileUpdate, repo *repo_model.Repository) ([]es.BulkOp, error) {
// Ignore vendored files in code search // Ignore vendored files in code search
if setting.Indexer.ExcludeVendored && analyze.IsVendor(update.Filename) { if setting.Indexer.ExcludeVendored && analyze.IsVendor(update.Filename) {
return nil, nil return nil, nil
@@ -157,8 +143,9 @@ func (b *Indexer) addUpdate(ctx context.Context, catFileBatch git.CatFileBatch,
} }
} }
id := internal.FilenameIndexerID(repo.ID, update.Filename)
if size > setting.Indexer.MaxIndexerFileSize { if size > setting.Indexer.MaxIndexerFileSize {
return []elastic.BulkableRequest{b.addDelete(update.Filename, repo)}, nil return []es.BulkOp{es.DeleteOp(id)}, nil
} }
info, batchReader, err := catFileBatch.QueryContent(update.BlobSha) info, batchReader, err := catFileBatch.QueryContent(update.BlobSha)
@@ -177,33 +164,24 @@ func (b *Indexer) addUpdate(ctx context.Context, catFileBatch git.CatFileBatch,
if _, err = batchReader.Discard(1); err != nil { if _, err = batchReader.Discard(1); err != nil {
return nil, err return nil, err
} }
id := internal.FilenameIndexerID(repo.ID, update.Filename)
return []elastic.BulkableRequest{ return []es.BulkOp{es.IndexOp(id, map[string]any{
elastic.NewBulkIndexRequest(). "repo_id": repo.ID,
Index(b.inner.VersionedIndexName()). "filename": update.Filename,
Id(id). "content": string(charset.ToUTF8DropErrors(fileContents)),
Doc(map[string]any{ "commit_id": sha,
"repo_id": repo.ID, "language": analyze.GetCodeLanguage(update.Filename, fileContents),
"filename": update.Filename, "updated_at": timeutil.TimeStampNow(),
"content": string(charset.ToUTF8DropErrors(fileContents)), })}, nil
"commit_id": sha,
"language": analyze.GetCodeLanguage(update.Filename, fileContents),
"updated_at": timeutil.TimeStampNow(),
}),
}, nil
} }
func (b *Indexer) addDelete(filename string, repo *repo_model.Repository) elastic.BulkableRequest { func (b *Indexer) addDelete(filename string, repo *repo_model.Repository) es.BulkOp {
id := internal.FilenameIndexerID(repo.ID, filename) return es.DeleteOp(internal.FilenameIndexerID(repo.ID, filename))
return elastic.NewBulkDeleteRequest().
Index(b.inner.VersionedIndexName()).
Id(id)
} }
// Index will save the index data // Index will save the index data
func (b *Indexer) Index(ctx context.Context, repo *repo_model.Repository, sha string, changes *internal.RepoChanges) error { func (b *Indexer) Index(ctx context.Context, repo *repo_model.Repository, sha string, changes *internal.RepoChanges) error {
reqs := make([]elastic.BulkableRequest, 0) ops := make([]es.BulkOp, 0)
if len(changes.Updates) > 0 { if len(changes.Updates) > 0 {
batch, err := gitrepo.NewBatch(ctx, repo) batch, err := gitrepo.NewBatch(ctx, repo)
if err != nil { if err != nil {
@@ -212,29 +190,25 @@ func (b *Indexer) Index(ctx context.Context, repo *repo_model.Repository, sha st
defer batch.Close() defer batch.Close()
for _, update := range changes.Updates { for _, update := range changes.Updates {
updateReqs, err := b.addUpdate(ctx, batch, sha, update, repo) updateOps, err := b.addUpdate(ctx, batch, sha, update, repo)
if err != nil { if err != nil {
return err return err
} }
if len(updateReqs) > 0 { if len(updateOps) > 0 {
reqs = append(reqs, updateReqs...) ops = append(ops, updateOps...)
} }
} }
} }
for _, filename := range changes.RemovedFilenames { for _, filename := range changes.RemovedFilenames {
reqs = append(reqs, b.addDelete(filename, repo)) ops = append(ops, b.addDelete(filename, repo))
} }
if len(reqs) > 0 { if len(ops) > 0 {
esBatchSize := 50 esBatchSize := 50
for i := 0; i < len(reqs); i += esBatchSize { for i := 0; i < len(ops); i += esBatchSize {
_, err := b.inner.Client.Bulk(). if err := b.Bulk(ctx, ops[i:min(i+esBatchSize, len(ops))]); err != nil {
Index(b.inner.VersionedIndexName()).
Add(reqs[i:min(i+esBatchSize, len(reqs))]...).
Do(ctx)
if err != nil {
return err return err
} }
} }
@@ -246,33 +220,21 @@ func (b *Indexer) Index(ctx context.Context, repo *repo_model.Repository, sha st
func (b *Indexer) Delete(ctx context.Context, repoID int64) error { func (b *Indexer) Delete(ctx context.Context, repoID int64) error {
if err := b.doDelete(ctx, repoID); err != nil { if err := b.doDelete(ctx, repoID); err != nil {
// Maybe there is a conflict during the delete operation, so we should retry after a refresh // Maybe there is a conflict during the delete operation, so we should retry after a refresh
log.Warn("Deletion of entries of repo %v within index %v was erroneous. Trying to refresh index before trying again", repoID, b.inner.VersionedIndexName(), err) log.Warn("Deletion of entries of repo %v within index %v was erroneous: %v. Trying to refresh index before trying again", repoID, b.VersionedIndexName(), err)
if err := b.refreshIndex(ctx); err != nil { if err := b.Refresh(ctx); err != nil {
return err return err
} }
if err := b.doDelete(ctx, repoID); err != nil { if err := b.doDelete(ctx, repoID); err != nil {
log.Error("Could not delete entries of repo %v within index %v", repoID, b.inner.VersionedIndexName()) log.Error("Could not delete entries of repo %v within index %v", repoID, b.VersionedIndexName())
return err return err
} }
} }
return nil return nil
} }
func (b *Indexer) refreshIndex(ctx context.Context) error {
if _, err := b.inner.Client.Refresh(b.inner.VersionedIndexName()).Do(ctx); err != nil {
log.Error("Error while trying to refresh index %v", b.inner.VersionedIndexName(), err)
return err
}
return nil
}
// Delete entries by repoId // Delete entries by repoId
func (b *Indexer) doDelete(ctx context.Context, repoID int64) error { func (b *Indexer) doDelete(ctx context.Context, repoID int64) error {
_, err := b.inner.Client.DeleteByQuery(b.inner.VersionedIndexName()). return b.DeleteByQuery(ctx, es.TermsQuery("repo_id", repoID))
Query(elastic.NewTermsQuery("repo_id", repoID)).
Do(ctx)
return err
} }
// contentMatchIndexPos find words positions for start and the following end on content. It will // contentMatchIndexPos find words positions for start and the following end on content. It will
@@ -291,10 +253,10 @@ func contentMatchIndexPos(content, start, end string) (int, int) {
return startIdx, (startIdx + len(start) + endIdx + len(end)) - 9 // remove the length <em></em> since we give Content the original data return startIdx, (startIdx + len(start) + endIdx + len(end)) - 9 // remove the length <em></em> since we give Content the original data
} }
func convertResult(searchResult *elastic.SearchResult, kw string, pageSize int) (int64, []*internal.SearchResult, []*internal.SearchResultLanguages, error) { func convertResult(searchResult *es.SearchResponse, kw string, pageSize int) (int64, []*internal.SearchResult, []*internal.SearchResultLanguages, error) {
hits := make([]*internal.SearchResult, 0, pageSize) hits := make([]*internal.SearchResult, 0, pageSize)
for _, hit := range searchResult.Hits.Hits { for _, hit := range searchResult.Hits {
repoID, fileName := internal.ParseIndexerID(hit.Id) repoID, fileName := internal.ParseIndexerID(hit.ID)
res := make(map[string]any) res := make(map[string]any)
if err := json.Unmarshal(hit.Source, &res); err != nil { if err := json.Unmarshal(hit.Source, &res); err != nil {
return 0, nil, nil, err return 0, nil, nil, err
@@ -333,111 +295,111 @@ func convertResult(searchResult *elastic.SearchResult, kw string, pageSize int)
}) })
} }
return searchResult.TotalHits(), hits, extractAggs(searchResult), nil return searchResult.Total, hits, extractAggs(searchResult), nil
} }
func extractAggs(searchResult *elastic.SearchResult) []*internal.SearchResultLanguages { func extractAggs(searchResult *es.SearchResponse) []*internal.SearchResultLanguages {
var searchResultLanguages []*internal.SearchResultLanguages buckets, found := searchResult.Aggregations["language"]
agg, found := searchResult.Aggregations.Terms("language") if !found {
if found { return nil
searchResultLanguages = make([]*internal.SearchResultLanguages, 0, 10) }
searchResultLanguages := make([]*internal.SearchResultLanguages, 0, 10)
for _, bucket := range agg.Buckets { for _, bucket := range buckets {
searchResultLanguages = append(searchResultLanguages, &internal.SearchResultLanguages{ // language is mapped as keyword so the key is always a string; if the
Language: bucket.Key.(string), // mapping ever changes, skip rather than emit an empty-language bucket.
Color: enry.GetColor(bucket.Key.(string)), key, ok := bucket.Key.(string)
Count: int(bucket.DocCount), if !ok {
}) continue
} }
searchResultLanguages = append(searchResultLanguages, &internal.SearchResultLanguages{
Language: key,
Color: enry.GetColor(key),
Count: int(bucket.DocCount),
})
} }
return searchResultLanguages return searchResultLanguages
} }
// Search searches for codes and language stats by given conditions. // Search searches for codes and language stats by given conditions.
func (b *Indexer) Search(ctx context.Context, opts *internal.SearchOptions) (int64, []*internal.SearchResult, []*internal.SearchResultLanguages, error) { func (b *Indexer) Search(ctx context.Context, opts *internal.SearchOptions) (int64, []*internal.SearchResult, []*internal.SearchResultLanguages, error) {
var contentQuery elastic.Query
searchMode := util.IfZero(opts.SearchMode, b.SupportedSearchModes()[0].ModeValue) searchMode := util.IfZero(opts.SearchMode, b.SupportedSearchModes()[0].ModeValue)
contentQuery := es.Query(es.NewMultiMatchQuery(opts.Keyword, "content").Type(es.MultiMatchTypeBestFields).Operator("and"))
if searchMode == indexer.SearchModeExact { if searchMode == indexer.SearchModeExact {
// 1.21 used NewMultiMatchQuery().Type(esMultiMatchTypePhrasePrefix), but later releases changed to NewMatchPhraseQuery contentQuery = es.MatchPhraseQuery("content", opts.Keyword)
contentQuery = elastic.NewMatchPhraseQuery("content", opts.Keyword)
} else /* words */ {
contentQuery = elastic.NewMultiMatchQuery("content", opts.Keyword).Type(esMultiMatchTypeBestFields).Operator("and")
} }
kwQuery := elastic.NewBoolQuery().Should( kwQuery := es.NewBoolQuery().Should(
contentQuery, contentQuery,
elastic.NewMultiMatchQuery(opts.Keyword, "filename^10").Type(esMultiMatchTypePhrasePrefix), es.NewMultiMatchQuery(opts.Keyword, "filename^10").Type(es.MultiMatchTypePhrasePrefix),
) )
query := elastic.NewBoolQuery() query := es.NewBoolQuery().Must(kwQuery)
query = query.Must(kwQuery)
if len(opts.RepoIDs) > 0 { if len(opts.RepoIDs) > 0 {
repoStrs := make([]any, 0, len(opts.RepoIDs)) query.Must(es.TermsQuery("repo_id", es.ToAnySlice(opts.RepoIDs)...))
for _, repoID := range opts.RepoIDs {
repoStrs = append(repoStrs, repoID)
}
repoQuery := elastic.NewTermsQuery("repo_id", repoStrs...)
query = query.Must(repoQuery)
} }
var ( start, pageSize := opts.GetSkipTake()
start, pageSize = opts.GetSkipTake() kw := "<em>" + opts.Keyword + "</em>"
kw = "<em>" + opts.Keyword + "</em>" languageAggs := map[string]any{
aggregation = elastic.NewTermsAggregation().Field("language").Size(10).OrderByCountDesc() "language": map[string]any{
) "terms": map[string]any{
"field": "language",
"size": 10,
"order": map[string]any{"_count": "desc"},
},
},
}
// number_of_fragments=0 returns the full highlighted content (no fragmentation).
highlight := map[string]any{
"fields": map[string]any{
"content": map[string]any{},
"filename": map[string]any{},
},
"number_of_fragments": 0,
"type": "fvh",
}
sort := []es.SortField{
{Field: "_score", Desc: true},
{Field: "updated_at", Desc: false},
}
if len(opts.Language) == 0 { if len(opts.Language) == 0 {
searchResult, err := b.inner.Client.Search(). resp, err := b.Indexer.Search(ctx, es.SearchRequest{
Index(b.inner.VersionedIndexName()). Query: query,
Aggregation("language", aggregation). Sort: sort,
Query(query). From: start,
Highlight( Size: pageSize,
elastic.NewHighlight(). TrackTotal: true,
Field("content"). Aggregations: languageAggs,
Field("filename"). Highlight: highlight,
NumOfFragments(0). // return all highlighting content on fragments })
HighlighterType("fvh"),
).
Sort("_score", false).
Sort("updated_at", true).
From(start).Size(pageSize).
Do(ctx)
if err != nil { if err != nil {
return 0, nil, nil, err return 0, nil, nil, err
} }
return convertResult(resp, kw, pageSize)
return convertResult(searchResult, kw, pageSize)
} }
langQuery := elastic.NewMatchQuery("language", opts.Language) countResp, err := b.Indexer.Search(ctx, es.SearchRequest{
countResult, err := b.inner.Client.Search(). Query: query,
Index(b.inner.VersionedIndexName()). Size: 0, // stats only
Aggregation("language", aggregation). TrackTotal: true,
Query(query). Aggregations: languageAggs,
Size(0). // We only need stats information })
Do(ctx)
if err != nil { if err != nil {
return 0, nil, nil, err return 0, nil, nil, err
} }
query = query.Must(langQuery) query.Must(es.MatchQuery("language", opts.Language))
searchResult, err := b.inner.Client.Search(). resp, err := b.Indexer.Search(ctx, es.SearchRequest{
Index(b.inner.VersionedIndexName()). Query: query,
Query(query). Sort: sort,
Highlight( From: start,
elastic.NewHighlight(). Size: pageSize,
Field("content"). TrackTotal: true,
Field("filename"). Highlight: highlight,
NumOfFragments(0). // return all highlighting content on fragments })
HighlighterType("fvh"),
).
Sort("_score", false).
Sort("updated_at", true).
From(start).Size(pageSize).
Do(ctx)
if err != nil { if err != nil {
return 0, nil, nil, err return 0, nil, nil, err
} }
total, hits, _, err := convertResult(searchResult, kw, pageSize) total, hits, _, err := convertResult(resp, kw, pageSize)
return total, hits, extractAggs(countResp), err
return total, hits, extractAggs(countResult), err
} }

View File

@@ -8,6 +8,7 @@ import (
"os" "os"
"slices" "slices"
"testing" "testing"
"time"
"code.gitea.io/gitea/models/db" "code.gitea.io/gitea/models/db"
"code.gitea.io/gitea/models/unittest" "code.gitea.io/gitea/models/unittest"
@@ -39,6 +40,16 @@ func TestMain(m *testing.M) {
func testIndexer(name string, t *testing.T, indexer internal.Indexer) { func testIndexer(name string, t *testing.T, indexer internal.Indexer) {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
assert.NoError(t, setupRepositoryIndexes(t.Context(), indexer)) assert.NoError(t, setupRepositoryIndexes(t.Context(), indexer))
// Wait for the index to catch up: ES/OpenSearch make writes visible
// only after a refresh (default interval: 1s). Bleve is synchronous
// and passes on the first iteration.
require.Eventually(t, func() bool {
total, _, _, err := indexer.Search(t.Context(), &internal.SearchOptions{
Keyword: "Description",
Paginator: &db.ListOptions{Page: 1, PageSize: 1},
})
return err == nil && total > 0
}, 10*time.Second, 100*time.Millisecond, "index did not become searchable")
keywords := []struct { keywords := []struct {
RepoIDs []int64 RepoIDs []int64

View File

@@ -4,52 +4,80 @@
package elasticsearch package elasticsearch
import ( import (
"bytes"
"context" "context"
"errors"
"fmt" "fmt"
"io"
"net"
"net/http"
"net/url"
"slices"
"strconv"
"strings"
"time"
"code.gitea.io/gitea/modules/indexer/internal" "code.gitea.io/gitea/modules/indexer/internal"
"code.gitea.io/gitea/modules/json"
"github.com/olivere/elastic/v7"
) )
var _ internal.Indexer = &Indexer{} var _ internal.Indexer = &Indexer{}
// Indexer represents a basic elasticsearch indexer implementation // Indexer is a narrow wrapper around an Elasticsearch/OpenSearch cluster.
// It targets the REST subset shared by Elasticsearch 7/8/9 and OpenSearch 3.
type Indexer struct { type Indexer struct {
Client *elastic.Client client *http.Client
base string // base URL with trailing slash, no userinfo
user string
pass string
url string
indexName string indexName string
version int version int
mapping string mapping string
} }
func NewIndexer(url, indexName string, version int, mapping string) *Indexer { // NewIndexer builds an Indexer. The connection is opened by Init.
func NewIndexer(rawURL, indexName string, version int, mapping string) *Indexer {
return &Indexer{ return &Indexer{
url: url, base: rawURL,
indexName: indexName, indexName: indexName,
version: version, version: version,
mapping: mapping, mapping: mapping,
} }
} }
// Init initializes the indexer // Init connects and creates the versioned index if missing, returning true if it already existed.
func (i *Indexer) Init(ctx context.Context) (bool, error) { func (i *Indexer) Init(ctx context.Context) (bool, error) {
if i == nil { parsed, err := url.Parse(i.base)
return false, errors.New("cannot init nil indexer")
}
if i.Client != nil {
return false, errors.New("indexer is already initialized")
}
client, err := i.initClient()
if err != nil { if err != nil {
return false, err return false, fmt.Errorf("parse elasticsearch url: %w", err)
}
if parsed.User != nil {
i.user = parsed.User.Username()
i.pass, _ = parsed.User.Password()
parsed.User = nil
}
base := parsed.String()
if !strings.HasSuffix(base, "/") {
base += "/"
}
i.base = base
// No client-level Timeout: bulk/_delete_by_query can legitimately run for
// minutes on large repos. Per-request deadlines come from the caller's ctx;
// transport-level timeouts cover stalled connects/handshakes/headers so a
// half-open server cannot wedge the indexer indefinitely.
i.client = &http.Client{
Transport: &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: (&net.Dialer{Timeout: 30 * time.Second, KeepAlive: 30 * time.Second}).DialContext,
TLSHandshakeTimeout: 10 * time.Second,
ResponseHeaderTimeout: 30 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
IdleConnTimeout: 90 * time.Second,
MaxIdleConns: 100,
},
} }
i.Client = client
exists, err := i.Client.IndexExists(i.VersionedIndexName()).Do(ctx) exists, err := i.indexExists(ctx, i.VersionedIndexName())
if err != nil { if err != nil {
return false, err return false, err
} }
@@ -61,34 +89,321 @@ func (i *Indexer) Init(ctx context.Context) (bool, error) {
return false, err return false, err
} }
return exists, nil return false, nil
} }
// Ping checks if the indexer is available // Ping returns an error when the cluster is unusable (status != green/yellow).
func (i *Indexer) Ping(ctx context.Context) error { func (i *Indexer) Ping(ctx context.Context) error {
if i == nil { var body struct {
return errors.New("cannot ping nil indexer") Status string `json:"status"`
} }
if i.Client == nil { if err := i.doJSON(ctx, http.MethodGet, "_cluster/health", nil, &body); err != nil {
return errors.New("indexer is not initialized")
}
resp, err := i.Client.ClusterHealth().Do(ctx)
if err != nil {
return err return err
} }
if resp.Status != "green" && resp.Status != "yellow" { // Healthy = green; usable = yellow. Red is unusable.
// It's healthy if the status is green, and it's available if the status is yellow, // https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html
// see https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html if body.Status != "green" && body.Status != "yellow" {
return fmt.Errorf("status of elasticsearch cluster is %s", resp.Status) return fmt.Errorf("status of elasticsearch cluster is %s", body.Status)
} }
return nil return nil
} }
// Close closes the indexer // Close releases idle HTTP connections held by the client.
func (i *Indexer) Close() { func (i *Indexer) Close() {
if i == nil { if i == nil || i.client == nil {
return return
} }
i.Client = nil i.client.CloseIdleConnections()
i.client = nil
}
// Bulk submits index/delete ops. Returns the first item-level failure, if any.
func (i *Indexer) Bulk(ctx context.Context, ops []BulkOp) error {
if len(ops) == 0 {
return nil
}
index := i.VersionedIndexName()
var buf bytes.Buffer
buf.Grow(len(ops) * 256)
for _, op := range ops {
meta := map[string]any{op.action: map[string]any{"_index": index, "_id": op.id}}
if err := writeJSONLine(&buf, meta); err != nil {
return err
}
if op.action == bulkActionIndex {
if err := writeJSONLine(&buf, op.doc); err != nil {
return err
}
}
}
res, err := i.do(ctx, http.MethodPost, urlPath(index, "_bulk"), "application/x-ndjson", bytes.NewReader(buf.Bytes()))
if err != nil {
return err
}
defer drainAndClose(res)
var body struct {
Errors bool `json:"errors"`
Items []map[string]struct {
Status int `json:"status"`
Error json.Value `json:"error"`
} `json:"items"`
}
if err := json.NewDecoder(res.Body).Decode(&body); err != nil {
return err
}
if !body.Errors {
return nil
}
return firstBulkError(body.Items)
}
// firstBulkError returns the first item-level failure in a bulk response.
// Each items entry is a single-key map ({"index": {...}} or {"delete": {...}}).
// Delete-of-missing (404) is idempotent and not reported.
func firstBulkError(items []map[string]struct {
Status int `json:"status"`
Error json.Value `json:"error"`
},
) error {
for _, item := range items {
for action, result := range item {
if action == bulkActionDelete && result.Status == http.StatusNotFound {
continue
}
if result.Status >= 300 {
return fmt.Errorf("bulk %s failed (status %d): %s", action, result.Status, string(result.Error))
}
}
}
return nil
}
// Index writes a single document.
func (i *Indexer) Index(ctx context.Context, id string, doc any) error {
body, err := json.Marshal(doc)
if err != nil {
return err
}
return i.doJSON(ctx, http.MethodPut, urlPath(i.VersionedIndexName(), "_doc", id), bytes.NewReader(body), nil)
}
// Delete removes a single document by id. Missing ids are not an error.
func (i *Indexer) Delete(ctx context.Context, id string) error {
res, err := i.do(ctx, http.MethodDelete, urlPath(i.VersionedIndexName(), "_doc", id), "", nil, http.StatusNotFound)
if err != nil {
return err
}
drainAndClose(res)
return nil
}
// DeleteByQuery removes every document matching the query.
func (i *Indexer) DeleteByQuery(ctx context.Context, query Query) error {
body, err := json.Marshal(map[string]any{"query": query.querySource()})
if err != nil {
return err
}
return i.doJSON(ctx, http.MethodPost, urlPath(i.VersionedIndexName(), "_delete_by_query"), bytes.NewReader(body), nil)
}
// Refresh forces a refresh so recent writes are searchable.
func (i *Indexer) Refresh(ctx context.Context) error {
return i.doJSON(ctx, http.MethodPost, urlPath(i.VersionedIndexName(), "_refresh"), nil, nil)
}
// Search runs a search request and decodes the reply.
func (i *Indexer) Search(ctx context.Context, req SearchRequest) (*SearchResponse, error) {
body := map[string]any{}
if req.Query != nil {
body["query"] = req.Query.querySource()
}
if len(req.Sort) > 0 {
sorts := make([]map[string]any, len(req.Sort))
for idx, s := range req.Sort {
sorts[idx] = s.source()
}
body["sort"] = sorts
}
if req.From > 0 {
body["from"] = req.From
}
body["size"] = req.Size
if len(req.Aggregations) > 0 {
body["aggs"] = req.Aggregations
}
if len(req.Highlight) > 0 {
body["highlight"] = req.Highlight
}
payload, err := json.Marshal(body)
if err != nil {
return nil, err
}
// Default track_total_hits is 10000 (capped count); send it explicitly so
// callers can choose between exact totals (true) and skipping counting (false).
path := urlPath(i.VersionedIndexName(), "_search") + "?track_total_hits=" + strconv.FormatBool(req.TrackTotal)
res, err := i.do(ctx, http.MethodPost, path, "application/json", bytes.NewReader(payload))
if err != nil {
return nil, err
}
defer drainAndClose(res)
return decodeSearchResponse(res.Body)
}
func (i *Indexer) indexExists(ctx context.Context, name string) (bool, error) {
res, err := i.do(ctx, http.MethodHead, urlPath(name), "", nil, http.StatusNotFound)
if err != nil {
return false, err
}
drainAndClose(res)
return res.StatusCode == http.StatusOK, nil
}
func (i *Indexer) createIndex(ctx context.Context) error {
var body struct {
Acknowledged bool `json:"acknowledged"`
}
if err := i.doJSON(ctx, http.MethodPut, urlPath(i.VersionedIndexName()), bytes.NewBufferString(i.mapping), &body); err != nil {
return fmt.Errorf("create index %s: %w", i.VersionedIndexName(), err)
}
if !body.Acknowledged {
return fmt.Errorf("create index %s not acknowledged", i.VersionedIndexName())
}
i.checkOldIndexes(ctx)
return nil
}
// do sends a request and returns the response. Status >= 300 is turned into
// an error unless the status appears in okStatus. The caller closes Body.
func (i *Indexer) do(ctx context.Context, method, path, contentType string, body io.Reader, okStatus ...int) (*http.Response, error) {
req, err := http.NewRequestWithContext(ctx, method, i.base+path, body)
if err != nil {
return nil, err
}
if contentType != "" {
req.Header.Set("Content-Type", contentType)
}
if i.user != "" || i.pass != "" {
req.SetBasicAuth(i.user, i.pass)
}
res, err := i.client.Do(req)
if err != nil {
return nil, err
}
if res.StatusCode >= 300 && !slices.Contains(okStatus, res.StatusCode) {
msg := readErrBody(res)
res.Body.Close()
return nil, fmt.Errorf("%s %s: %s", method, path, msg)
}
return res, nil
}
// doJSON sends a request with a JSON body and, when out is non-nil, decodes
// the JSON response into it.
func (i *Indexer) doJSON(ctx context.Context, method, path string, body io.Reader, out any) error {
contentType := ""
if body != nil {
contentType = "application/json"
}
res, err := i.do(ctx, method, path, contentType, body)
if err != nil {
return err
}
defer drainAndClose(res)
if out == nil {
return nil
}
return json.NewDecoder(res.Body).Decode(out)
}
// drainAndClose discards any unread response body before closing so the
// underlying TCP connection can be reused for keep-alive.
func drainAndClose(res *http.Response) {
_, _ = io.Copy(io.Discard, res.Body)
res.Body.Close()
}
func writeJSONLine(buf *bytes.Buffer, v any) error {
enc, err := json.Marshal(v)
if err != nil {
return err
}
buf.Write(enc)
buf.WriteByte('\n')
return nil
}
// readErrBody reads up to 4 KiB of an error response and drains the rest so
// the underlying connection can be reused (keep-alive needs Body fully read).
func readErrBody(res *http.Response) string {
const limit = 4 << 10
b, _ := io.ReadAll(io.LimitReader(res.Body, limit))
_, _ = io.Copy(io.Discard, res.Body)
return fmt.Sprintf("status %d: %s", res.StatusCode, bytes.TrimSpace(b))
}
func decodeSearchResponse(r io.Reader) (*SearchResponse, error) {
var raw struct {
Hits struct {
Total struct {
Value int64 `json:"value"`
} `json:"total"`
Hits []struct {
ID string `json:"_id"`
Score float64 `json:"_score"`
Source json.Value `json:"_source"`
Highlight map[string][]string `json:"highlight"`
} `json:"hits"`
} `json:"hits"`
Aggregations map[string]struct {
Buckets []struct {
Key any `json:"key"`
DocCount int64 `json:"doc_count"`
} `json:"buckets"`
} `json:"aggregations"`
}
if err := json.NewDecoder(r).Decode(&raw); err != nil {
return nil, err
}
resp := &SearchResponse{
Total: raw.Hits.Total.Value,
Hits: make([]SearchHit, 0, len(raw.Hits.Hits)),
}
for _, h := range raw.Hits.Hits {
resp.Hits = append(resp.Hits, SearchHit{
ID: h.ID,
Score: h.Score,
Source: h.Source,
Highlight: h.Highlight,
})
}
if len(raw.Aggregations) > 0 {
resp.Aggregations = make(map[string][]AggBucket, len(raw.Aggregations))
for name, agg := range raw.Aggregations {
buckets := make([]AggBucket, len(agg.Buckets))
for idx, b := range agg.Buckets {
buckets[idx] = AggBucket{Key: b.Key, DocCount: b.DocCount}
}
resp.Aggregations[name] = buckets
}
}
return resp, nil
}
// urlPath joins path segments with `/` and percent-escapes each.
func urlPath(segments ...string) string {
var b bytes.Buffer
for idx, s := range segments {
if idx > 0 {
b.WriteByte('/')
}
b.WriteString(url.PathEscape(s))
}
return b.String()
} }

View File

@@ -0,0 +1,44 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package elasticsearch
import (
"os"
"strings"
"testing"
"github.com/stretchr/testify/require"
)
func newRealIndexer(t *testing.T) *Indexer {
t.Helper()
url := "http://elasticsearch:9200"
if os.Getenv("CI") == "" {
url = os.Getenv("TEST_ELASTICSEARCH_URL")
if url == "" {
t.Skip("TEST_ELASTICSEARCH_URL not set and not running in CI")
}
}
indexName := "gitea_test_" + strings.ReplaceAll(strings.ToLower(t.Name()), "/", "_")
ix := NewIndexer(url, indexName, 1, `{"mappings":{"properties":{"x":{"type":"keyword"}}}}`)
_, err := ix.Init(t.Context())
require.NoError(t, err)
t.Cleanup(ix.Close)
return ix
}
func TestPing(t *testing.T) {
ix := newRealIndexer(t)
require.NoError(t, ix.Ping(t.Context()))
}
func TestDeleteSwallows404(t *testing.T) {
ix := newRealIndexer(t)
require.NoError(t, ix.Delete(t.Context(), "missing-id"))
}
func TestBulkAcceptsDelete404(t *testing.T) {
ix := newRealIndexer(t)
require.NoError(t, ix.Bulk(t.Context(), []BulkOp{DeleteOp("missing-id")}))
}

View File

@@ -0,0 +1,132 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package elasticsearch
// MultiMatch types used by the call sites. See
// https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-multi-match-query.html#multi-match-types
const (
MultiMatchTypeBestFields = "best_fields"
MultiMatchTypePhrasePrefix = "phrase_prefix"
)
// ToAnySlice converts []T to []any for variadic query args like TermsQuery.
func ToAnySlice[T any](s []T) []any {
out := make([]any, len(s))
for idx, v := range s {
out[idx] = v
}
return out
}
// Query is an Elasticsearch query DSL node. It marshals to the JSON
// object expected by the ES query API.
type Query interface {
querySource() map[string]any
}
type rawQuery map[string]any
func (q rawQuery) querySource() map[string]any { return q }
// TermQuery matches documents whose `field` exactly equals `value`.
func TermQuery(field string, value any) Query {
return rawQuery{"term": map[string]any{field: value}}
}
// TermsQuery matches documents whose `field` equals any of `values`.
func TermsQuery(field string, values ...any) Query {
return rawQuery{"terms": map[string]any{field: values}}
}
// MatchQuery is a full-text match on a single field.
func MatchQuery(field string, value any) Query {
return rawQuery{"match": map[string]any{field: value}}
}
// MatchPhraseQuery matches the exact phrase on `field`.
func MatchPhraseQuery(field, value string) Query {
return rawQuery{"match_phrase": map[string]any{field: value}}
}
// MultiMatchQuery is the fluent builder for a multi_match query.
type MultiMatchQuery struct {
query any
fields []string
typ string
operator string
}
// NewMultiMatchQuery creates a multi_match query over the given fields.
func NewMultiMatchQuery(query any, fields ...string) *MultiMatchQuery {
return &MultiMatchQuery{query: query, fields: fields}
}
func (m *MultiMatchQuery) Type(t string) *MultiMatchQuery { m.typ = t; return m }
func (m *MultiMatchQuery) Operator(op string) *MultiMatchQuery { m.operator = op; return m }
func (m *MultiMatchQuery) querySource() map[string]any {
body := map[string]any{"query": m.query}
if len(m.fields) > 0 {
body["fields"] = m.fields
}
if m.typ != "" {
body["type"] = m.typ
}
if m.operator != "" {
body["operator"] = m.operator
}
return map[string]any{"multi_match": body}
}
// RangeQuery is the fluent builder for a range query.
type RangeQuery struct {
field string
body map[string]any
}
func NewRangeQuery(field string) *RangeQuery {
return &RangeQuery{field: field, body: map[string]any{}}
}
func (r *RangeQuery) Gte(v any) *RangeQuery { r.body["gte"] = v; return r }
func (r *RangeQuery) Lte(v any) *RangeQuery { r.body["lte"] = v; return r }
func (r *RangeQuery) querySource() map[string]any {
return map[string]any{"range": map[string]any{r.field: r.body}}
}
// BoolQuery is the fluent builder for a bool query.
type BoolQuery struct {
must []Query
should []Query
mustNot []Query
}
func NewBoolQuery() *BoolQuery { return &BoolQuery{} }
func (b *BoolQuery) Must(q ...Query) *BoolQuery { b.must = append(b.must, q...); return b }
func (b *BoolQuery) Should(q ...Query) *BoolQuery { b.should = append(b.should, q...); return b }
func (b *BoolQuery) MustNot(q ...Query) *BoolQuery { b.mustNot = append(b.mustNot, q...); return b }
func (b *BoolQuery) querySource() map[string]any {
body := map[string]any{}
if len(b.must) > 0 {
body["must"] = querySlice(b.must)
}
if len(b.should) > 0 {
body["should"] = querySlice(b.should)
}
if len(b.mustNot) > 0 {
body["must_not"] = querySlice(b.mustNot)
}
return map[string]any{"bool": body}
}
func querySlice(queries []Query) []map[string]any {
out := make([]map[string]any, len(queries))
for idx, q := range queries {
out[idx] = q.querySource()
}
return out
}

View File

@@ -0,0 +1,76 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package elasticsearch
import "code.gitea.io/gitea/modules/json"
const (
bulkActionIndex = "index"
bulkActionDelete = "delete"
)
// BulkOp is a single write inside a Bulk call. Construct with IndexOp or DeleteOp.
type BulkOp struct {
action string
id string
doc any
}
// IndexOp builds a bulk index operation.
func IndexOp(id string, doc any) BulkOp {
return BulkOp{action: bulkActionIndex, id: id, doc: doc}
}
// DeleteOp builds a bulk delete operation.
func DeleteOp(id string) BulkOp {
return BulkOp{action: bulkActionDelete, id: id}
}
// SortField is one entry of the search sort array.
type SortField struct {
Field string
Desc bool
}
func (s SortField) source() map[string]any {
order := "asc"
if s.Desc {
order = "desc"
}
return map[string]any{s.Field: map[string]any{"order": order}}
}
// SearchRequest captures everything Gitea sends to the _search endpoint.
// Aggregations and Highlight are raw ES JSON bodies — callers write them as
// map[string]any since each has exactly one call site with a fixed shape.
type SearchRequest struct {
Query Query
Sort []SortField
From int
Size int
TrackTotal bool
Aggregations map[string]any
Highlight map[string]any
}
// SearchHit is a single result row.
type SearchHit struct {
ID string
Score float64
Source json.Value
Highlight map[string][]string
}
// AggBucket is a terms-aggregation bucket.
type AggBucket struct {
Key any
DocCount int64
}
// SearchResponse is Gitea's decoded view of the search reply.
type SearchResponse struct {
Total int64
Hits []SearchHit
Aggregations map[string][]AggBucket
}

View File

@@ -6,14 +6,11 @@ package elasticsearch
import ( import (
"context" "context"
"fmt" "fmt"
"time"
"code.gitea.io/gitea/modules/log" "code.gitea.io/gitea/modules/log"
"github.com/olivere/elastic/v7"
) )
// VersionedIndexName returns the full index name with version // VersionedIndexName returns the full index name with version suffix.
func (i *Indexer) VersionedIndexName() string { func (i *Indexer) VersionedIndexName() string {
return versionedIndexName(i.indexName, i.version) return versionedIndexName(i.indexName, i.version)
} }
@@ -26,41 +23,10 @@ func versionedIndexName(indexName string, version int) string {
return fmt.Sprintf("%s.v%d", indexName, version) return fmt.Sprintf("%s.v%d", indexName, version)
} }
func (i *Indexer) createIndex(ctx context.Context) error {
createIndex, err := i.Client.CreateIndex(i.VersionedIndexName()).BodyString(i.mapping).Do(ctx)
if err != nil {
return err
}
if !createIndex.Acknowledged {
return fmt.Errorf("create index %s with %s failed", i.VersionedIndexName(), i.mapping)
}
i.checkOldIndexes(ctx)
return nil
}
func (i *Indexer) initClient() (*elastic.Client, error) {
opts := []elastic.ClientOptionFunc{
elastic.SetURL(i.url),
elastic.SetSniff(false),
elastic.SetHealthcheckInterval(10 * time.Second),
elastic.SetGzip(false),
}
logger := log.GetLogger(log.DEFAULT)
opts = append(opts, elastic.SetTraceLog(&log.PrintfLogger{Logf: logger.Trace}))
opts = append(opts, elastic.SetInfoLog(&log.PrintfLogger{Logf: logger.Info}))
opts = append(opts, elastic.SetErrorLog(&log.PrintfLogger{Logf: logger.Error}))
return elastic.NewClient(opts...)
}
func (i *Indexer) checkOldIndexes(ctx context.Context) { func (i *Indexer) checkOldIndexes(ctx context.Context) {
for v := 0; v < i.version; v++ { for v := range i.version {
indexName := versionedIndexName(i.indexName, v) indexName := versionedIndexName(i.indexName, v)
exists, err := i.Client.IndexExists(indexName).Do(ctx) exists, err := i.indexExists(ctx, indexName)
if err == nil && exists { if err == nil && exists {
log.Warn("Found older elasticsearch index named %q, Gitea will keep the old NOT DELETED. You can delete the old version after the upgrade succeed.", indexName) log.Warn("Found older elasticsearch index named %q, Gitea will keep the old NOT DELETED. You can delete the old version after the upgrade succeed.", indexName)
} }

View File

@@ -27,7 +27,7 @@ import (
const ( const (
issueIndexerAnalyzer = "issueIndexer" issueIndexerAnalyzer = "issueIndexer"
issueIndexerDocType = "issueIndexerDocType" issueIndexerDocType = "issueIndexerDocType"
issueIndexerLatestVersion = 5 issueIndexerLatestVersion = 6
) )
const unicodeNormalizeName = "unicodeNormalize" const unicodeNormalizeName = "unicodeNormalize"
@@ -83,8 +83,8 @@ func generateIssueIndexMapping() (mapping.IndexMapping, error) {
docMapping.AddFieldMappingsAt("label_ids", numberFieldMapping) docMapping.AddFieldMappingsAt("label_ids", numberFieldMapping)
docMapping.AddFieldMappingsAt("no_label", boolFieldMapping) docMapping.AddFieldMappingsAt("no_label", boolFieldMapping)
docMapping.AddFieldMappingsAt("milestone_id", numberFieldMapping) docMapping.AddFieldMappingsAt("milestone_id", numberFieldMapping)
docMapping.AddFieldMappingsAt("project_id", numberFieldMapping) docMapping.AddFieldMappingsAt("project_ids", numberFieldMapping)
docMapping.AddFieldMappingsAt("project_board_id", numberFieldMapping) docMapping.AddFieldMappingsAt("no_project", boolFieldMapping)
docMapping.AddFieldMappingsAt("poster_id", numberFieldMapping) docMapping.AddFieldMappingsAt("poster_id", numberFieldMapping)
docMapping.AddFieldMappingsAt("assignee_id", numberFieldMapping) docMapping.AddFieldMappingsAt("assignee_id", numberFieldMapping)
docMapping.AddFieldMappingsAt("mention_ids", numberFieldMapping) docMapping.AddFieldMappingsAt("mention_ids", numberFieldMapping)
@@ -241,11 +241,15 @@ func (b *Indexer) Search(ctx context.Context, options *internal.SearchOptions) (
queries = append(queries, bleve.NewDisjunctionQuery(milestoneQueries...)) queries = append(queries, bleve.NewDisjunctionQuery(milestoneQueries...))
} }
if options.ProjectID.Has() { if options.NoProjectOnly {
queries = append(queries, inner_bleve.NumericEqualityQuery(options.ProjectID.Value(), "project_id")) queries = append(queries, inner_bleve.BoolFieldQuery(true, "no_project"))
} } else if len(options.ProjectIDs) > 0 {
if options.ProjectColumnID.Has() { var projectQueries []query.Query
queries = append(queries, inner_bleve.NumericEqualityQuery(options.ProjectColumnID.Value(), "project_board_id")) for _, projectID := range options.ProjectIDs {
projectQueries = append(projectQueries, inner_bleve.NumericEqualityQuery(projectID, "project_ids"))
}
// FIXME: ISSUE-MULTIPLE-PROJECTS-FILTER: this logic is not right, it should use "AND" but not "OR"
queries = append(queries, bleve.NewDisjunctionQuery(projectQueries...))
} }
if options.PosterID != "" { if options.PosterID != "" {

View File

@@ -13,6 +13,7 @@ import (
"code.gitea.io/gitea/modules/container" "code.gitea.io/gitea/modules/container"
"code.gitea.io/gitea/modules/indexer/issues/internal" "code.gitea.io/gitea/modules/indexer/issues/internal"
"code.gitea.io/gitea/modules/optional" "code.gitea.io/gitea/modules/optional"
"code.gitea.io/gitea/modules/util"
) )
func ToDBOptions(ctx context.Context, options *internal.SearchOptions) (*issue_model.IssuesOptions, error) { func ToDBOptions(ctx context.Context, options *internal.SearchOptions) (*issue_model.IssuesOptions, error) {
@@ -65,8 +66,7 @@ func ToDBOptions(ctx context.Context, options *internal.SearchOptions) (*issue_m
ReviewRequestedID: convertID(options.ReviewRequestedID), ReviewRequestedID: convertID(options.ReviewRequestedID),
ReviewedID: convertID(options.ReviewedID), ReviewedID: convertID(options.ReviewedID),
SubscriberID: convertID(options.SubscriberID), SubscriberID: convertID(options.SubscriberID),
ProjectID: convertID(options.ProjectID), ProjectIDs: util.Iif(options.NoProjectOnly, []int64{db.NoConditionID}, options.ProjectIDs),
ProjectColumnID: convertID(options.ProjectColumnID),
IsClosed: options.IsClosed, IsClosed: options.IsClosed,
IsPull: options.IsPull, IsPull: options.IsPull,
IncludedLabelNames: nil, IncludedLabelNames: nil,

View File

@@ -46,10 +46,10 @@ func ToSearchOptions(keyword string, opts *issues_model.IssuesOptions) *SearchOp
searchOpt.MilestoneIDs = opts.MilestoneIDs searchOpt.MilestoneIDs = opts.MilestoneIDs
} }
if opts.ProjectID > 0 { if len(opts.ProjectIDs) == 1 && opts.ProjectIDs[0] == db.NoConditionID {
searchOpt.ProjectID = optional.Some(opts.ProjectID) searchOpt.NoProjectOnly = true
} else if opts.ProjectID == db.NoConditionID { // FIXME: this is inconsistent from other places } else {
searchOpt.ProjectID = optional.Some[int64](0) // Those issues with no project(projectid==0) searchOpt.ProjectIDs = opts.ProjectIDs
} }
searchOpt.AssigneeID = opts.AssigneeID searchOpt.AssigneeID = opts.AssigneeID
@@ -65,7 +65,6 @@ func ToSearchOptions(keyword string, opts *issues_model.IssuesOptions) *SearchOp
return nil return nil
} }
searchOpt.ProjectColumnID = convertID(opts.ProjectColumnID)
searchOpt.PosterID = opts.PosterID searchOpt.PosterID = opts.PosterID
searchOpt.MentionID = convertID(opts.MentionedID) searchOpt.MentionID = convertID(opts.MentionedID)
searchOpt.ReviewedID = convertID(opts.ReviewedID) searchOpt.ReviewedID = convertID(opts.ReviewedID)

View File

@@ -11,27 +11,18 @@ import (
"code.gitea.io/gitea/modules/graceful" "code.gitea.io/gitea/modules/graceful"
"code.gitea.io/gitea/modules/indexer" "code.gitea.io/gitea/modules/indexer"
indexer_internal "code.gitea.io/gitea/modules/indexer/internal" indexer_internal "code.gitea.io/gitea/modules/indexer/internal"
inner_elasticsearch "code.gitea.io/gitea/modules/indexer/internal/elasticsearch" es "code.gitea.io/gitea/modules/indexer/internal/elasticsearch"
"code.gitea.io/gitea/modules/indexer/issues/internal" "code.gitea.io/gitea/modules/indexer/issues/internal"
"code.gitea.io/gitea/modules/util" "code.gitea.io/gitea/modules/util"
"github.com/olivere/elastic/v7"
) )
const ( const issueIndexerLatestVersion = 3
issueIndexerLatestVersion = 2
// multi-match-types, currently only 2 types are used
// Reference: https://www.elastic.co/guide/en/elasticsearch/reference/7.0/query-dsl-multi-match-query.html#multi-match-types
esMultiMatchTypeBestFields = "best_fields"
esMultiMatchTypePhrasePrefix = "phrase_prefix"
)
var _ internal.Indexer = &Indexer{} var _ internal.Indexer = &Indexer{}
// Indexer implements Indexer interface // Indexer implements Indexer interface
type Indexer struct { type Indexer struct {
inner *inner_elasticsearch.Indexer *es.Indexer
indexer_internal.Indexer // do not composite inner_elasticsearch.Indexer directly to avoid exposing too much
} }
func (b *Indexer) SupportedSearchModes() []indexer.SearchMode { func (b *Indexer) SupportedSearchModes() []indexer.SearchMode {
@@ -41,12 +32,7 @@ func (b *Indexer) SupportedSearchModes() []indexer.SearchMode {
// NewIndexer creates a new elasticsearch indexer // NewIndexer creates a new elasticsearch indexer
func NewIndexer(url, indexerName string) *Indexer { func NewIndexer(url, indexerName string) *Indexer {
inner := inner_elasticsearch.NewIndexer(url, indexerName, issueIndexerLatestVersion, defaultMapping) return &Indexer{Indexer: es.NewIndexer(url, indexerName, issueIndexerLatestVersion, defaultMapping)}
indexer := &Indexer{
inner: inner,
Indexer: inner,
}
return indexer
} }
const ( const (
@@ -68,8 +54,8 @@ const (
"label_ids": { "type": "integer", "index": true }, "label_ids": { "type": "integer", "index": true },
"no_label": { "type": "boolean", "index": true }, "no_label": { "type": "boolean", "index": true },
"milestone_id": { "type": "integer", "index": true }, "milestone_id": { "type": "integer", "index": true },
"project_id": { "type": "integer", "index": true }, "project_ids": { "type": "integer", "index": true },
"project_board_id": { "type": "integer", "index": true }, "no_project": { "type": "boolean", "index": true },
"poster_id": { "type": "integer", "index": true }, "poster_id": { "type": "integer", "index": true },
"assignee_id": { "type": "integer", "index": true }, "assignee_id": { "type": "integer", "index": true },
"mention_ids": { "type": "integer", "index": true }, "mention_ids": { "type": "integer", "index": true },
@@ -93,29 +79,14 @@ func (b *Indexer) Index(ctx context.Context, issues ...*internal.IndexerData) er
return nil return nil
} else if len(issues) == 1 { } else if len(issues) == 1 {
issue := issues[0] issue := issues[0]
_, err := b.inner.Client.Index(). return b.Indexer.Index(ctx, strconv.FormatInt(issue.ID, 10), issue)
Index(b.inner.VersionedIndexName()).
Id(strconv.FormatInt(issue.ID, 10)).
BodyJson(issue).
Do(ctx)
return err
} }
reqs := make([]elastic.BulkableRequest, 0) ops := make([]es.BulkOp, 0, len(issues))
for _, issue := range issues { for _, issue := range issues {
reqs = append(reqs, ops = append(ops, es.IndexOp(strconv.FormatInt(issue.ID, 10), issue))
elastic.NewBulkIndexRequest().
Index(b.inner.VersionedIndexName()).
Id(strconv.FormatInt(issue.ID, 10)).
Doc(issue),
)
} }
return b.Bulk(graceful.GetManager().HammerContext(), ops)
_, err := b.inner.Client.Bulk().
Index(b.inner.VersionedIndexName()).
Add(reqs...).
Do(graceful.GetManager().HammerContext())
return err
} }
// Delete deletes indexes by ids // Delete deletes indexes by ids
@@ -123,129 +94,116 @@ func (b *Indexer) Delete(ctx context.Context, ids ...int64) error {
if len(ids) == 0 { if len(ids) == 0 {
return nil return nil
} else if len(ids) == 1 { } else if len(ids) == 1 {
_, err := b.inner.Client.Delete(). return b.Indexer.Delete(ctx, strconv.FormatInt(ids[0], 10))
Index(b.inner.VersionedIndexName()).
Id(strconv.FormatInt(ids[0], 10)).
Do(ctx)
return err
} }
reqs := make([]elastic.BulkableRequest, 0) ops := make([]es.BulkOp, 0, len(ids))
for _, id := range ids { for _, id := range ids {
reqs = append(reqs, ops = append(ops, es.DeleteOp(strconv.FormatInt(id, 10)))
elastic.NewBulkDeleteRequest().
Index(b.inner.VersionedIndexName()).
Id(strconv.FormatInt(id, 10)),
)
} }
return b.Bulk(graceful.GetManager().HammerContext(), ops)
_, err := b.inner.Client.Bulk().
Index(b.inner.VersionedIndexName()).
Add(reqs...).
Do(graceful.GetManager().HammerContext())
return err
} }
// Search searches for issues by given conditions. // Search searches for issues by given conditions.
// Returns the matching issue IDs // Returns the matching issue IDs
func (b *Indexer) Search(ctx context.Context, options *internal.SearchOptions) (*internal.SearchResult, error) { func (b *Indexer) Search(ctx context.Context, options *internal.SearchOptions) (*internal.SearchResult, error) {
query := elastic.NewBoolQuery() query := es.NewBoolQuery()
if options.Keyword != "" { if options.Keyword != "" {
searchMode := util.IfZero(options.SearchMode, b.SupportedSearchModes()[0].ModeValue) searchMode := util.IfZero(options.SearchMode, b.SupportedSearchModes()[0].ModeValue)
mm := es.NewMultiMatchQuery(options.Keyword, "title", "content", "comments")
if searchMode == indexer.SearchModeExact { if searchMode == indexer.SearchModeExact {
query.Must(elastic.NewMultiMatchQuery(options.Keyword, "title", "content", "comments").Type(esMultiMatchTypePhrasePrefix)) mm = mm.Type(es.MultiMatchTypePhrasePrefix)
} else /* words */ { } else {
query.Must(elastic.NewMultiMatchQuery(options.Keyword, "title", "content", "comments").Type(esMultiMatchTypeBestFields).Operator("and")) mm = mm.Type(es.MultiMatchTypeBestFields).Operator("and")
} }
query.Must(mm)
} }
if len(options.RepoIDs) > 0 { if len(options.RepoIDs) > 0 {
q := elastic.NewBoolQuery() q := es.NewBoolQuery()
q.Should(elastic.NewTermsQuery("repo_id", toAnySlice(options.RepoIDs)...)) q.Should(es.TermsQuery("repo_id", es.ToAnySlice(options.RepoIDs)...))
if options.AllPublic { if options.AllPublic {
q.Should(elastic.NewTermQuery("is_public", true)) q.Should(es.TermQuery("is_public", true))
} }
query.Must(q) query.Must(q)
} }
if options.IsPull.Has() { if options.IsPull.Has() {
query.Must(elastic.NewTermQuery("is_pull", options.IsPull.Value())) query.Must(es.TermQuery("is_pull", options.IsPull.Value()))
} }
if options.IsClosed.Has() { if options.IsClosed.Has() {
query.Must(elastic.NewTermQuery("is_closed", options.IsClosed.Value())) query.Must(es.TermQuery("is_closed", options.IsClosed.Value()))
} }
if options.IsArchived.Has() { if options.IsArchived.Has() {
query.Must(elastic.NewTermQuery("is_archived", options.IsArchived.Value())) query.Must(es.TermQuery("is_archived", options.IsArchived.Value()))
} }
if options.NoLabelOnly { if options.NoLabelOnly {
query.Must(elastic.NewTermQuery("no_label", true)) query.Must(es.TermQuery("no_label", true))
} else { } else {
if len(options.IncludedLabelIDs) > 0 { if len(options.IncludedLabelIDs) > 0 {
q := elastic.NewBoolQuery() q := es.NewBoolQuery()
for _, labelID := range options.IncludedLabelIDs { for _, labelID := range options.IncludedLabelIDs {
q.Must(elastic.NewTermQuery("label_ids", labelID)) q.Must(es.TermQuery("label_ids", labelID))
} }
query.Must(q) query.Must(q)
} else if len(options.IncludedAnyLabelIDs) > 0 { } else if len(options.IncludedAnyLabelIDs) > 0 {
query.Must(elastic.NewTermsQuery("label_ids", toAnySlice(options.IncludedAnyLabelIDs)...)) query.Must(es.TermsQuery("label_ids", es.ToAnySlice(options.IncludedAnyLabelIDs)...))
} }
if len(options.ExcludedLabelIDs) > 0 { if len(options.ExcludedLabelIDs) > 0 {
q := elastic.NewBoolQuery() q := es.NewBoolQuery()
for _, labelID := range options.ExcludedLabelIDs { for _, labelID := range options.ExcludedLabelIDs {
q.MustNot(elastic.NewTermQuery("label_ids", labelID)) q.MustNot(es.TermQuery("label_ids", labelID))
} }
query.Must(q) query.Must(q)
} }
} }
if len(options.MilestoneIDs) > 0 { if len(options.MilestoneIDs) > 0 {
query.Must(elastic.NewTermsQuery("milestone_id", toAnySlice(options.MilestoneIDs)...)) query.Must(es.TermsQuery("milestone_id", es.ToAnySlice(options.MilestoneIDs)...))
} }
if options.ProjectID.Has() { if options.NoProjectOnly {
query.Must(elastic.NewTermQuery("project_id", options.ProjectID.Value())) query.Must(es.TermQuery("no_project", true))
} } else if len(options.ProjectIDs) > 0 {
if options.ProjectColumnID.Has() { // FIXME: ISSUE-MULTIPLE-PROJECTS-FILTER: this logic is not right, it should use "AND" but not "OR"
query.Must(elastic.NewTermQuery("project_board_id", options.ProjectColumnID.Value())) query.Must(es.TermsQuery("project_ids", es.ToAnySlice(options.ProjectIDs)...))
} }
if options.PosterID != "" { if options.PosterID != "" {
// "(none)" becomes 0, it means no poster // "(none)" becomes 0, it means no poster
posterIDInt64, _ := strconv.ParseInt(options.PosterID, 10, 64) posterIDInt64, _ := strconv.ParseInt(options.PosterID, 10, 64)
query.Must(elastic.NewTermQuery("poster_id", posterIDInt64)) query.Must(es.TermQuery("poster_id", posterIDInt64))
} }
if options.AssigneeID != "" { if options.AssigneeID != "" {
if options.AssigneeID == "(any)" { if options.AssigneeID == "(any)" {
q := elastic.NewRangeQuery("assignee_id") query.Must(es.NewRangeQuery("assignee_id").Gte(1))
q.Gte(1)
query.Must(q)
} else { } else {
// "(none)" becomes 0, it means no assignee // "(none)" becomes 0, it means no assignee
assigneeIDInt64, _ := strconv.ParseInt(options.AssigneeID, 10, 64) assigneeIDInt64, _ := strconv.ParseInt(options.AssigneeID, 10, 64)
query.Must(elastic.NewTermQuery("assignee_id", assigneeIDInt64)) query.Must(es.TermQuery("assignee_id", assigneeIDInt64))
} }
} }
if options.MentionID.Has() { if options.MentionID.Has() {
query.Must(elastic.NewTermQuery("mention_ids", options.MentionID.Value())) query.Must(es.TermQuery("mention_ids", options.MentionID.Value()))
} }
if options.ReviewedID.Has() { if options.ReviewedID.Has() {
query.Must(elastic.NewTermQuery("reviewed_ids", options.ReviewedID.Value())) query.Must(es.TermQuery("reviewed_ids", options.ReviewedID.Value()))
} }
if options.ReviewRequestedID.Has() { if options.ReviewRequestedID.Has() {
query.Must(elastic.NewTermQuery("review_requested_ids", options.ReviewRequestedID.Value())) query.Must(es.TermQuery("review_requested_ids", options.ReviewRequestedID.Value()))
} }
if options.SubscriberID.Has() { if options.SubscriberID.Has() {
query.Must(elastic.NewTermQuery("subscriber_ids", options.SubscriberID.Value())) query.Must(es.TermQuery("subscriber_ids", options.SubscriberID.Value()))
} }
if options.UpdatedAfterUnix.Has() || options.UpdatedBeforeUnix.Has() { if options.UpdatedAfterUnix.Has() || options.UpdatedBeforeUnix.Has() {
q := elastic.NewRangeQuery("updated_unix") q := es.NewRangeQuery("updated_unix")
if options.UpdatedAfterUnix.Has() { if options.UpdatedAfterUnix.Has() {
q.Gte(options.UpdatedAfterUnix.Value()) q.Gte(options.UpdatedAfterUnix.Value())
} }
@@ -258,9 +216,9 @@ func (b *Indexer) Search(ctx context.Context, options *internal.SearchOptions) (
if options.SortBy == "" { if options.SortBy == "" {
options.SortBy = internal.SortByCreatedAsc options.SortBy = internal.SortByCreatedAsc
} }
sortBy := []elastic.Sorter{ sortBy := []es.SortField{
parseSortBy(options.SortBy), parseSortBy(options.SortBy),
elastic.NewFieldSort("id").Desc(), {Field: "id", Desc: true},
} }
// See https://stackoverflow.com/questions/35206409/elasticsearch-2-1-result-window-is-too-large-index-max-result-window/35221900 // See https://stackoverflow.com/questions/35206409/elasticsearch-2-1-result-window-is-too-large-index-max-result-window/35221900
@@ -268,43 +226,30 @@ func (b *Indexer) Search(ctx context.Context, options *internal.SearchOptions) (
const maxPageSize = 10000 const maxPageSize = 10000
skip, limit := indexer_internal.ParsePaginator(options.Paginator, maxPageSize) skip, limit := indexer_internal.ParsePaginator(options.Paginator, maxPageSize)
searchResult, err := b.inner.Client.Search(). resp, err := b.Indexer.Search(ctx, es.SearchRequest{
Index(b.inner.VersionedIndexName()). Query: query,
Query(query). Sort: sortBy,
SortBy(sortBy...). From: skip,
From(skip).Size(limit). Size: limit,
Do(ctx) TrackTotal: true,
})
if err != nil { if err != nil {
return nil, err return nil, err
} }
hits := make([]internal.Match, 0, limit) hits := make([]internal.Match, 0, len(resp.Hits))
for _, hit := range searchResult.Hits.Hits { for _, hit := range resp.Hits {
id, _ := strconv.ParseInt(hit.Id, 10, 64) id, _ := strconv.ParseInt(hit.ID, 10, 64)
hits = append(hits, internal.Match{ hits = append(hits, internal.Match{ID: id})
ID: id,
})
} }
return &internal.SearchResult{ return &internal.SearchResult{
Total: searchResult.TotalHits(), Total: resp.Total,
Hits: hits, Hits: hits,
}, nil }, nil
} }
func toAnySlice[T any](s []T) []any { func parseSortBy(sortBy internal.SortBy) es.SortField {
ret := make([]any, 0, len(s)) field, desc := strings.CutPrefix(string(sortBy), "-")
for _, item := range s { return es.SortField{Field: field, Desc: desc}
ret = append(ret, item)
}
return ret
}
func parseSortBy(sortBy internal.SortBy) elastic.Sorter {
field := strings.TrimPrefix(string(sortBy), "-")
ret := elastic.NewFieldSort(field)
if strings.HasPrefix(string(sortBy), "-") {
ret.Desc()
}
return ret
} }

View File

@@ -6,6 +6,7 @@ package elasticsearch
import ( import (
"fmt" "fmt"
"net/http" "net/http"
"net/url"
"os" "os"
"testing" "testing"
"time" "time"
@@ -17,19 +18,36 @@ import (
func TestElasticsearchIndexer(t *testing.T) { func TestElasticsearchIndexer(t *testing.T) {
// The elasticsearch instance started by pull-db-tests.yml > test-unit > services > elasticsearch // The elasticsearch instance started by pull-db-tests.yml > test-unit > services > elasticsearch
url := "http://elastic:changeme@elasticsearch:9200" rawURL := "http://elastic:changeme@elasticsearch:9200"
if os.Getenv("CI") == "" { if os.Getenv("CI") == "" {
// Make it possible to run tests against a local elasticsearch instance // Make it possible to run tests against a local elasticsearch instance
url = os.Getenv("TEST_ELASTICSEARCH_URL") rawURL = os.Getenv("TEST_ELASTICSEARCH_URL")
if url == "" { if rawURL == "" {
t.Skip("TEST_ELASTICSEARCH_URL not set and not running in CI") t.Skip("TEST_ELASTICSEARCH_URL not set and not running in CI")
return return
} }
} }
// Go's net/http does not auto-attach URL userinfo as Basic Auth, so extract
// it and set the header explicitly; otherwise auth-enforced clusters answer
// 401 and the probe never reports ready.
parsed, err := url.Parse(rawURL)
require.NoError(t, err)
user := parsed.User
parsed.User = nil
probeURL := parsed.String()
require.Eventually(t, func() bool { require.Eventually(t, func() bool {
resp, err := http.Get(url) req, err := http.NewRequest(http.MethodGet, probeURL, nil)
if err != nil {
return false
}
if user != nil {
pass, _ := user.Password()
req.SetBasicAuth(user.Username(), pass)
}
resp, err := http.DefaultClient.Do(req)
if err != nil { if err != nil {
return false return false
} }
@@ -37,7 +55,7 @@ func TestElasticsearchIndexer(t *testing.T) {
return resp.StatusCode == http.StatusOK return resp.StatusCode == http.StatusOK
}, time.Minute, time.Second, "Expected elasticsearch to be up") }, time.Minute, time.Second, "Expected elasticsearch to be up")
indexer := NewIndexer(url, fmt.Sprintf("test_elasticsearch_indexer_%d", time.Now().Unix())) indexer := NewIndexer(rawURL, fmt.Sprintf("test_elasticsearch_indexer_%d", time.Now().Unix()))
defer indexer.Close() defer indexer.Close()
tests.TestIndexer(t, indexer) tests.TestIndexer(t, indexer)

View File

@@ -416,28 +416,42 @@ func searchIssueInProject(t *testing.T) {
}{ }{
{ {
SearchOptions{ SearchOptions{
ProjectID: optional.Some(int64(1)), ProjectIDs: []int64{1},
}, },
[]int64{5, 3, 2, 1}, []int64{5, 3, 2, 1},
}, },
{
SearchOptions{
ProjectColumnID: optional.Some(int64(1)),
},
[]int64{1},
},
{
SearchOptions{
ProjectColumnID: optional.Some(int64(0)), // issue with in default column
},
[]int64{2},
},
} }
for _, test := range tests { for _, test := range tests {
issueIDs, _, err := SearchIssues(t.Context(), &test.opts) issueIDs, _, err := SearchIssues(t.Context(), &test.opts)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, test.expectedIDs, issueIDs) assert.Equal(t, test.expectedIDs, issueIDs)
} }
// Test filtering for issues with no project assigned using dynamic validation
t.Run("no project assigned", func(t *testing.T) {
issueIDs, total, err := SearchIssues(t.Context(), &SearchOptions{
ProjectIDs: []int64{db.NoConditionID},
})
require.NoError(t, err)
assert.NotEmpty(t, issueIDs)
assert.Equal(t, total, int64(len(issueIDs)))
// Verify each returned issue actually has no project
for _, issueID := range issueIDs {
issue, err := issues.GetIssueByID(t.Context(), issueID)
require.NoError(t, err)
err = issue.LoadProjects(t.Context())
require.NoError(t, err)
assert.Empty(t, issue.Projects, "Issue %d should have no projects", issueID)
}
// Count total issues with no project to verify we got them all
allIssues, err := issues.Issues(t.Context(), &issues.IssuesOptions{
ProjectIDs: []int64{db.NoConditionID},
})
require.NoError(t, err)
assert.Len(t, issueIDs, len(allIssues), "Should return all issues with no project")
})
} }
func searchIssueWithPaginator(t *testing.T) { func searchIssueWithPaginator(t *testing.T) {

View File

@@ -30,8 +30,9 @@ type IndexerData struct {
LabelIDs []int64 `json:"label_ids"` LabelIDs []int64 `json:"label_ids"`
NoLabel bool `json:"no_label"` // True if LabelIDs is empty NoLabel bool `json:"no_label"` // True if LabelIDs is empty
MilestoneID int64 `json:"milestone_id"` MilestoneID int64 `json:"milestone_id"`
ProjectID int64 `json:"project_id"` ProjectIDs []int64 `json:"project_ids"`
ProjectColumnID int64 `json:"project_board_id"` // the key should be kept as project_board_id to keep compatible NoProject bool `json:"no_project"` // True if ProjectIDs is empty
ProjectColumnMap map[int64]int64 `json:"project_column_map,omitempty"` // Maps project ID to column ID for each project the issue is in
PosterID int64 `json:"poster_id"` PosterID int64 `json:"poster_id"`
AssigneeID int64 `json:"assignee_id"` AssigneeID int64 `json:"assignee_id"`
MentionIDs []int64 `json:"mention_ids"` MentionIDs []int64 `json:"mention_ids"`
@@ -94,8 +95,8 @@ type SearchOptions struct {
MilestoneIDs []int64 // milestones the issues have MilestoneIDs []int64 // milestones the issues have
ProjectID optional.Option[int64] // project the issues belong to ProjectIDs []int64 // project the issues belong to. FIXME: ISSUE-MULTIPLE-PROJECTS-FILTER: no multiple project filter support yet. Search logic is wrong.
ProjectColumnID optional.Option[int64] // project column the issues belong to NoProjectOnly bool // if the issues have no project, if true, ProjectIDs will be ignored
PosterID string // poster of the issues, "(none)" or "(any)" or a user ID PosterID string // poster of the issues, "(none)" or "(any)" or a user ID
AssigneeID string // assignee of the issues, "(none)" or "(any)" or a user ID AssigneeID string // assignee of the issues, "(none)" or "(any)" or a user ID

View File

@@ -116,6 +116,16 @@ var cases = []*testIndexerCase{
assert.Equal(t, len(data), int(result.Total)) assert.Equal(t, len(data), int(result.Total))
}, },
}, },
{
// Exercises the single-doc Index/Delete fast path in backends that have one (e.g. Elasticsearch).
Name: "single-doc index",
ExtraData: []*internal.IndexerData{
{ID: 999, Title: "solo-issue-marker"},
},
SearchOptions: &internal.SearchOptions{Keyword: "solo-issue-marker"},
ExpectedIDs: []int64{999},
ExpectedTotal: 1,
},
{ {
Name: "Keyword", Name: "Keyword",
ExtraData: []*internal.IndexerData{ ExtraData: []*internal.IndexerData{
@@ -301,75 +311,41 @@ var cases = []*testIndexerCase{
}, },
}, },
{ {
Name: "ProjectID", Name: "ProjectIDs",
SearchOptions: &internal.SearchOptions{ SearchOptions: &internal.SearchOptions{
Paginator: &db.ListOptions{ Paginator: &db.ListOptions{
PageSize: 5, PageSize: 5,
}, },
ProjectID: optional.Some(int64(1)), ProjectIDs: []int64{1},
}, },
Expected: func(t *testing.T, data map[int64]*internal.IndexerData, result *internal.SearchResult) { Expected: func(t *testing.T, data map[int64]*internal.IndexerData, result *internal.SearchResult) {
assert.Len(t, result.Hits, 5) assert.Len(t, result.Hits, 5)
for _, v := range result.Hits { for _, v := range result.Hits {
assert.Equal(t, int64(1), data[v.ID].ProjectID) assert.Contains(t, data[v.ID].ProjectIDs, int64(1))
} }
assert.Equal(t, countIndexerData(data, func(v *internal.IndexerData) bool { assert.Equal(t, countIndexerData(data, func(v *internal.IndexerData) bool {
return v.ProjectID == 1 return slices.Contains(v.ProjectIDs, int64(1))
}), result.Total) }), result.Total)
}, },
}, },
{ {
Name: "no ProjectID", Name: "no ProjectIDs (empty array)",
SearchOptions: &internal.SearchOptions{ SearchOptions: &internal.SearchOptions{
Paginator: &db.ListOptions{ Paginator: &db.ListOptions{
PageSize: 5, PageSize: 50,
}, },
ProjectID: optional.Some(int64(0)), NoProjectOnly: true,
}, },
Expected: func(t *testing.T, data map[int64]*internal.IndexerData, result *internal.SearchResult) { Expected: func(t *testing.T, data map[int64]*internal.IndexerData, result *internal.SearchResult) {
assert.Len(t, result.Hits, 5) // Verify only issues with no projects are returned
for _, v := range result.Hits { for _, v := range result.Hits {
assert.Equal(t, int64(0), data[v.ID].ProjectID) assert.Empty(t, data[v.ID].ProjectIDs, "Issue %d should have no projects", v.ID)
} }
assert.Equal(t, countIndexerData(data, func(v *internal.IndexerData) bool { // Verify we got ALL issues with no projects
return v.ProjectID == 0 expectedCount := countIndexerData(data, func(v *internal.IndexerData) bool {
}), result.Total) return len(v.ProjectIDs) == 0
}, })
}, assert.Equal(t, expectedCount, result.Total, "Should return all %d issues with no project", expectedCount)
{
Name: "ProjectColumnID",
SearchOptions: &internal.SearchOptions{
Paginator: &db.ListOptions{
PageSize: 5,
},
ProjectColumnID: optional.Some(int64(1)),
},
Expected: func(t *testing.T, data map[int64]*internal.IndexerData, result *internal.SearchResult) {
assert.Len(t, result.Hits, 5)
for _, v := range result.Hits {
assert.Equal(t, int64(1), data[v.ID].ProjectColumnID)
}
assert.Equal(t, countIndexerData(data, func(v *internal.IndexerData) bool {
return v.ProjectColumnID == 1
}), result.Total)
},
},
{
Name: "no ProjectColumnID",
SearchOptions: &internal.SearchOptions{
Paginator: &db.ListOptions{
PageSize: 5,
},
ProjectColumnID: optional.Some(int64(0)),
},
Expected: func(t *testing.T, data map[int64]*internal.IndexerData, result *internal.SearchResult) {
assert.Len(t, result.Hits, 5)
for _, v := range result.Hits {
assert.Equal(t, int64(0), data[v.ID].ProjectColumnID)
}
assert.Equal(t, countIndexerData(data, func(v *internal.IndexerData) bool {
return v.ProjectColumnID == 0
}), result.Total)
}, },
}, },
{ {
@@ -706,6 +682,10 @@ func generateDefaultIndexerData() []*internal.IndexerData {
for i := range subscriberIDs { for i := range subscriberIDs {
subscriberIDs[i] = int64(i) + 1 // SubscriberID should not be 0 subscriberIDs[i] = int64(i) + 1 // SubscriberID should not be 0
} }
projectIDs := make([]int64, id%5)
for i := range projectIDs {
projectIDs[i] = int64(i) + 1 // projectID should not be 0
}
data = append(data, &internal.IndexerData{ data = append(data, &internal.IndexerData{
ID: id, ID: id,
@@ -719,8 +699,8 @@ func generateDefaultIndexerData() []*internal.IndexerData {
LabelIDs: labelIDs, LabelIDs: labelIDs,
NoLabel: len(labelIDs) == 0, NoLabel: len(labelIDs) == 0,
MilestoneID: issueIndex % 4, MilestoneID: issueIndex % 4,
ProjectID: issueIndex % 5, ProjectIDs: projectIDs,
ProjectColumnID: issueIndex % 6, NoProject: len(projectIDs) == 0,
PosterID: id%10 + 1, // PosterID should not be 0 PosterID: id%10 + 1, // PosterID should not be 0
AssigneeID: issueIndex % 10, AssigneeID: issueIndex % 10,
MentionIDs: mentionIDs, MentionIDs: mentionIDs,

View File

@@ -20,7 +20,7 @@ import (
) )
const ( const (
issueIndexerLatestVersion = 4 issueIndexerLatestVersion = 5
// TODO: make this configurable if necessary // TODO: make this configurable if necessary
maxTotalHits = 10000 maxTotalHits = 10000
@@ -71,8 +71,8 @@ func NewIndexer(url, apiKey, indexerName string) *Indexer {
"label_ids", "label_ids",
"no_label", "no_label",
"milestone_id", "milestone_id",
"project_id", "project_ids",
"project_board_id", "no_project",
"poster_id", "poster_id",
"assignee_id", "assignee_id",
"mention_ids", "mention_ids",
@@ -182,11 +182,11 @@ func (b *Indexer) Search(ctx context.Context, options *internal.SearchOptions) (
query.And(inner_meilisearch.NewFilterIn("milestone_id", options.MilestoneIDs...)) query.And(inner_meilisearch.NewFilterIn("milestone_id", options.MilestoneIDs...))
} }
if options.ProjectID.Has() { if options.NoProjectOnly {
query.And(inner_meilisearch.NewFilterEq("project_id", options.ProjectID.Value())) query.And(inner_meilisearch.NewFilterEq("no_project", true))
} } else if len(options.ProjectIDs) > 0 {
if options.ProjectColumnID.Has() { // FIXME: ISSUE-MULTIPLE-PROJECTS-FILTER: this logic is not right, it should use "AND" but not "OR"
query.And(inner_meilisearch.NewFilterEq("project_board_id", options.ProjectColumnID.Value())) query.And(inner_meilisearch.NewFilterIn("project_ids", options.ProjectIDs...))
} }
if options.PosterID != "" { if options.PosterID != "" {

View File

@@ -87,14 +87,9 @@ func getIssueIndexerData(ctx context.Context, issueID int64) (*internal.IndexerD
return nil, false, err return nil, false, err
} }
var projectID int64 projectIDs := make([]int64, 0, len(issue.Projects))
if issue.Project != nil { for _, project := range issue.Projects {
projectID = issue.Project.ID projectIDs = append(projectIDs, project.ID)
}
projectColumnID, err := issue.ProjectColumnID(ctx)
if err != nil {
return nil, false, err
} }
if err := issue.Repo.LoadOwner(ctx); err != nil { if err := issue.Repo.LoadOwner(ctx); err != nil {
@@ -114,8 +109,8 @@ func getIssueIndexerData(ctx context.Context, issueID int64) (*internal.IndexerD
LabelIDs: labels, LabelIDs: labels,
NoLabel: len(labels) == 0, NoLabel: len(labels) == 0,
MilestoneID: issue.MilestoneID, MilestoneID: issue.MilestoneID,
ProjectID: projectID, ProjectIDs: projectIDs,
ProjectColumnID: projectColumnID, NoProject: len(projectIDs) == 0,
PosterID: issue.PosterID, PosterID: issue.PosterID,
AssigneeID: issue.AssigneeID, AssigneeID: issue.AssigneeID,
MentionIDs: mentionIDs, MentionIDs: mentionIDs,

View File

@@ -4,13 +4,7 @@
package setting package setting
import ( import (
"errors"
"fmt"
"net"
"net/url"
"os"
"path/filepath" "path/filepath"
"strings"
"time" "time"
) )
@@ -20,24 +14,22 @@ var (
// DatabaseTypeNames contains the friendly names for all database types // DatabaseTypeNames contains the friendly names for all database types
DatabaseTypeNames = map[string]string{"mysql": "MySQL", "postgres": "PostgreSQL", "mssql": "MSSQL", "sqlite3": "SQLite3"} DatabaseTypeNames = map[string]string{"mysql": "MySQL", "postgres": "PostgreSQL", "mssql": "MSSQL", "sqlite3": "SQLite3"}
// EnableSQLite3 use SQLite3, set by build flag
EnableSQLite3 bool
// Database holds the database settings // Database holds the database settings
Database = struct { Database = struct {
Type DatabaseType Type DatabaseType
Host string Host string
Name string Name string
User string User string
Passwd string Passwd string
Schema string Schema string
SSLMode string SSLMode string
Path string Path string
SQLiteBusyTimeout int
SQLiteJournalMode string
LogSQL bool LogSQL bool
MysqlCharset string
CharsetCollation string CharsetCollation string
Timeout int // seconds
SQLiteJournalMode string
DBConnectRetries int DBConnectRetries int
DBConnectBackoff time.Duration DBConnectBackoff time.Duration
MaxIdleConns int MaxIdleConns int
@@ -47,7 +39,7 @@ var (
AutoMigration bool AutoMigration bool
SlowQueryThreshold time.Duration SlowQueryThreshold time.Duration
}{ }{
Timeout: 500, SQLiteBusyTimeout: 500,
IterateBufferSize: 50, IterateBufferSize: 50,
} }
) )
@@ -64,15 +56,14 @@ func loadDBSetting(rootCfg ConfigProvider) {
Database.Host = sec.Key("HOST").String() Database.Host = sec.Key("HOST").String()
Database.Name = sec.Key("NAME").String() Database.Name = sec.Key("NAME").String()
Database.User = sec.Key("USER").String() Database.User = sec.Key("USER").String()
if len(Database.Passwd) == 0 { Database.Passwd = sec.Key("PASSWD").String()
Database.Passwd = sec.Key("PASSWD").String()
}
Database.Schema = sec.Key("SCHEMA").String() Database.Schema = sec.Key("SCHEMA").String()
Database.SSLMode = sec.Key("SSL_MODE").MustString("disable") Database.SSLMode = sec.Key("SSL_MODE").MustString("disable")
Database.CharsetCollation = sec.Key("CHARSET_COLLATION").String() Database.CharsetCollation = sec.Key("CHARSET_COLLATION").String()
Database.Path = sec.Key("PATH").MustString(filepath.Join(AppDataPath, "gitea.db")) Database.Path = sec.Key("PATH").MustString(filepath.Join(AppDataPath, "gitea.db"))
Database.Timeout = sec.Key("SQLITE_TIMEOUT").MustInt(500) Database.SQLiteBusyTimeout = sec.Key("SQLITE_TIMEOUT").MustInt(500)
Database.SQLiteJournalMode = sec.Key("SQLITE_JOURNAL_MODE").MustString("") Database.SQLiteJournalMode = sec.Key("SQLITE_JOURNAL_MODE").MustString("")
Database.MaxIdleConns = sec.Key("MAX_IDLE_CONNS").MustInt(2) Database.MaxIdleConns = sec.Key("MAX_IDLE_CONNS").MustInt(2)
@@ -91,123 +82,9 @@ func loadDBSetting(rootCfg ConfigProvider) {
Database.SlowQueryThreshold = sec.Key("SLOW_QUERY_THRESHOLD").MustDuration(5 * time.Second) Database.SlowQueryThreshold = sec.Key("SLOW_QUERY_THRESHOLD").MustDuration(5 * time.Second)
} }
// DBConnStr returns database connection string // DatabaseType FIXME: it is also used directly with "schemas.DBType", so the names must be consistent
func DBConnStr() (string, error) {
var connStr string
paramSep := "?"
if strings.Contains(Database.Name, paramSep) {
paramSep = "&"
}
switch Database.Type {
case "mysql":
connType := "tcp"
if len(Database.Host) > 0 && Database.Host[0] == '/' { // looks like a unix socket
connType = "unix"
}
tls := Database.SSLMode
if tls == "disable" { // allow (Postgres-inspired) default value to work in MySQL
tls = "false"
}
connStr = fmt.Sprintf("%s:%s@%s(%s)/%s%sparseTime=true&tls=%s",
Database.User, Database.Passwd, connType, Database.Host, Database.Name, paramSep, tls)
case "postgres":
connStr = getPostgreSQLConnectionString(Database.Host, Database.User, Database.Passwd, Database.Name, Database.SSLMode)
case "mssql":
host, port := ParseMSSQLHostPort(Database.Host)
connStr = fmt.Sprintf("server=%s; port=%s; database=%s; user id=%s; password=%s;", host, port, Database.Name, Database.User, Database.Passwd)
case "sqlite3":
if !EnableSQLite3 {
return "", errors.New("this Gitea binary was not built with SQLite3 support")
}
if err := os.MkdirAll(filepath.Dir(Database.Path), os.ModePerm); err != nil {
return "", fmt.Errorf("Failed to create directories: %w", err)
}
journalMode := ""
if Database.SQLiteJournalMode != "" {
journalMode = "&_journal_mode=" + Database.SQLiteJournalMode
}
connStr = fmt.Sprintf("file:%s?cache=shared&mode=rwc&_busy_timeout=%d&_txlock=immediate%s",
Database.Path, Database.Timeout, journalMode)
default:
return "", fmt.Errorf("unknown database type: %s", Database.Type)
}
return connStr, nil
}
// parsePostgreSQLHostPort parses given input in various forms defined in
// https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-CONNSTRING
// and returns proper host and port number.
func parsePostgreSQLHostPort(info string) (host, port string) {
if h, p, err := net.SplitHostPort(info); err == nil {
host, port = h, p
} else {
// treat the "info" as "host", if it's an IPv6 address, remove the wrapper
host = info
if strings.HasPrefix(host, "[") && strings.HasSuffix(host, "]") {
host = host[1 : len(host)-1]
}
}
// set fallback values
if host == "" {
host = "127.0.0.1"
}
if port == "" {
port = "5432"
}
return host, port
}
func getPostgreSQLConnectionString(dbHost, dbUser, dbPasswd, dbName, dbsslMode string) (connStr string) {
dbName, dbParam, _ := strings.Cut(dbName, "?")
host, port := parsePostgreSQLHostPort(dbHost)
connURL := url.URL{
Scheme: "postgres",
User: url.UserPassword(dbUser, dbPasswd),
Host: net.JoinHostPort(host, port),
Path: dbName,
OmitHost: false,
RawQuery: dbParam,
}
query := connURL.Query()
if strings.HasPrefix(host, "/") { // looks like a unix socket
query.Add("host", host)
connURL.Host = ":" + port
}
query.Set("sslmode", dbsslMode)
connURL.RawQuery = query.Encode()
return connURL.String()
}
// ParseMSSQLHostPort splits the host into host and port
func ParseMSSQLHostPort(info string) (string, string) {
// the default port "0" might be related to MSSQL's dynamic port, maybe it should be double-confirmed in the future
host, port := "127.0.0.1", "0"
if strings.Contains(info, ":") {
host = strings.Split(info, ":")[0]
port = strings.Split(info, ":")[1]
} else if strings.Contains(info, ",") {
host = strings.Split(info, ",")[0]
port = strings.TrimSpace(strings.Split(info, ",")[1])
} else if len(info) > 0 {
host = info
}
if host == "" {
host = "127.0.0.1"
}
if port == "" {
port = "0"
}
return host, port
}
type DatabaseType string type DatabaseType string
func (t DatabaseType) String() string {
return string(t)
}
func (t DatabaseType) IsSQLite3() bool { func (t DatabaseType) IsSQLite3() bool {
return t == "sqlite3" return t == "sqlite3"
} }

View File

@@ -1,15 +0,0 @@
//go:build sqlite
// Copyright 2014 The Gogs Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package setting
import (
_ "github.com/mattn/go-sqlite3"
)
func init() {
EnableSQLite3 = true
SupportedDatabaseTypes = append(SupportedDatabaseTypes, "sqlite3")
}

View File

@@ -60,6 +60,7 @@ type Issue struct {
Attachments []*Attachment `json:"assets"` Attachments []*Attachment `json:"assets"`
Labels []*Label `json:"labels"` Labels []*Label `json:"labels"`
Milestone *Milestone `json:"milestone"` Milestone *Milestone `json:"milestone"`
Projects []*Project `json:"projects"`
// deprecated // deprecated
Assignee *User `json:"assignee"` Assignee *User `json:"assignee"`
Assignees []*User `json:"assignees"` Assignees []*User `json:"assignees"`
@@ -100,7 +101,9 @@ type CreateIssueOption struct {
Milestone int64 `json:"milestone"` Milestone int64 `json:"milestone"`
// list of label ids // list of label ids
Labels []int64 `json:"labels"` Labels []int64 `json:"labels"`
Closed bool `json:"closed"` // list of project ids
Projects []int64 `json:"projects"`
Closed bool `json:"closed"`
} }
// EditIssueOption options for editing an issue // EditIssueOption options for editing an issue
@@ -112,7 +115,9 @@ type EditIssueOption struct {
Assignee *string `json:"assignee"` Assignee *string `json:"assignee"`
Assignees []string `json:"assignees"` Assignees []string `json:"assignees"`
Milestone *int64 `json:"milestone"` Milestone *int64 `json:"milestone"`
State *string `json:"state"` // list of project ids to set (replaces existing projects)
Projects *[]int64 `json:"projects"`
State *string `json:"state"`
// swagger:strfmt date-time // swagger:strfmt date-time
Deadline *time.Time `json:"due_date"` Deadline *time.Time `json:"due_date"`
RemoveDeadline *bool `json:"unset_due_date"` RemoveDeadline *bool `json:"unset_due_date"`

View File

@@ -0,0 +1,33 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package structs
import (
"time"
)
// Project represents a project
// swagger:model
type Project struct {
// ID is the unique identifier for the project
ID int64 `json:"id"`
// Title is the title of the project
Title string `json:"title"`
// Description provides details about the project
Description string `json:"description"`
// OwnerID is the owner of the project (for org-level projects)
OwnerID int64 `json:"owner_id,omitempty"`
// RepoID is the repository this project belongs to (for repo-level projects)
RepoID int64 `json:"repo_id,omitempty"`
// CreatorID is the user who created the project
CreatorID int64 `json:"creator_id"`
// IsClosed indicates if the project is closed
IsClosed bool `json:"is_closed"`
// swagger:strfmt date-time
Created time.Time `json:"created_at"`
// swagger:strfmt date-time
Updated time.Time `json:"updated_at"`
// swagger:strfmt date-time
Closed *time.Time `json:"closed_at,omitempty"`
}

View File

@@ -236,6 +236,12 @@ type EditRepoOption struct {
MirrorInterval *string `json:"mirror_interval,omitempty"` MirrorInterval *string `json:"mirror_interval,omitempty"`
// enable prune - remove obsolete remote-tracking references when mirroring // enable prune - remove obsolete remote-tracking references when mirroring
EnablePrune *bool `json:"enable_prune,omitempty"` EnablePrune *bool `json:"enable_prune,omitempty"`
// authentication username for the remote repository (mirrors)
MirrorUsername *string `json:"mirror_username,omitempty"`
// authentication password for the remote repository (mirrors)
MirrorPassword *string `json:"mirror_password,omitempty"`
// authentication token for the remote repository (mirrors)
MirrorToken *string `json:"mirror_token,omitempty"`
} }
// GenerateRepoOption options when creating a repository using a template // GenerateRepoOption options when creating a repository using a template

View File

@@ -5,6 +5,7 @@ package templates
import ( import (
"context" "context"
"fmt"
"html/template" "html/template"
"io" "io"
"net/http" "net/http"
@@ -36,7 +37,11 @@ func (r *pageRenderer) funcMapDummy() template.FuncMap {
} }
func (r *pageRenderer) TemplateLookup(tmpl string, templateCtx context.Context) (TemplateExecutor, error) { //nolint:revive // we don't use ctx, only pass it to the template executor func (r *pageRenderer) TemplateLookup(tmpl string, templateCtx context.Context) (TemplateExecutor, error) { //nolint:revive // we don't use ctx, only pass it to the template executor
return r.tmplRenderer.Templates().Executor(tmpl, r.funcMap(templateCtx)) tmpls := r.tmplRenderer.Templates()
if tmpls == nil {
return nil, fmt.Errorf("no templates defined for %s", tmpl)
}
return tmpls.Executor(tmpl, r.funcMap(templateCtx))
} }
func (r *pageRenderer) HTML(w io.Writer, status int, tplName TplName, data any, templateCtx context.Context) error { //nolint:revive // we don't use ctx, only pass it to the template executor func (r *pageRenderer) HTML(w io.Writer, status int, tplName TplName, data any, templateCtx context.Context) error { //nolint:revive // we don't use ctx, only pass it to the template executor

View File

@@ -6,6 +6,11 @@ package templates
import ( import (
"fmt" "fmt"
"reflect" "reflect"
"slices"
"strconv"
"strings"
"code.gitea.io/gitea/modules/util"
) )
type SliceUtils struct{} type SliceUtils struct{}
@@ -33,3 +38,29 @@ func (su *SliceUtils) Contains(s, v any) bool {
} }
return false return false
} }
// JoinInt64 joins a slice of int64 values into a comma-separated string.
func (su *SliceUtils) JoinInt64(values []int64) string {
if len(values) == 0 {
return ""
}
strs := make([]string, len(values))
for i, v := range values {
strs[i] = strconv.FormatInt(v, 10)
}
return strings.Join(strs, ",")
}
func (su *SliceUtils) JoinToggleIDs(values []int64, target int64) (ret struct {
IsIncluded bool
ToggledIDs string
},
) {
ret.IsIncluded = slices.Contains(values, target)
if ret.IsIncluded {
ret.ToggledIDs = su.JoinInt64(util.SliceRemoveAll(slices.Clone(values), target))
} else {
ret.ToggledIDs = su.JoinInt64(append(values, target))
}
return ret
}

View File

@@ -70,6 +70,16 @@ func TestUtils(t *testing.T) {
actual = execTmpl("{{StringUtils.Contains .String .Value}}", map[string]any{"String": "abc", "Value": "x"}) actual = execTmpl("{{StringUtils.Contains .String .Value}}", map[string]any{"String": "abc", "Value": "x"})
assert.Equal(t, "false", actual) assert.Equal(t, "false", actual)
// Test JoinInt64
actual = execTmpl("{{SliceUtils.JoinInt64 .Values}}", map[string]any{"Values": []int64{1, 2, 3}})
assert.Equal(t, "1,2,3", actual)
actual = execTmpl("{{SliceUtils.JoinInt64 .Values}}", map[string]any{"Values": []int64{}})
assert.Empty(t, actual)
actual = execTmpl("{{SliceUtils.JoinInt64 .Values}}", map[string]any{"Values": []int64{42}})
assert.Equal(t, "42", actual)
tmpl := template.New("test") tmpl := template.New("test")
tmpl.Funcs(template.FuncMap{"SliceUtils": NewSliceUtils, "StringUtils": NewStringUtils}) tmpl.Funcs(template.FuncMap{"SliceUtils": NewSliceUtils, "StringUtils": NewStringUtils})
template.Must(tmpl.Parse("{{SliceUtils.Contains .Slice .Value}}")) template.Must(tmpl.Parse("{{SliceUtils.Contains .Slice .Value}}"))

Some files were not shown because too many files have changed in this diff Show More